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Abstract 


The  Aeronautical  System  Center  (ASC)  is  developing  a  Simulation  and  Analysis 
Facility  (SIMAF)  that  will  link  models,  simulations,  hardware-in-the-loop,  and  system-in- 
the-loop  resources  to  create  a  robust  virtual  environment  supporting  assessment  of 
alternate  systems  in  the  defense  acquisition  process.  ENS  is  assisting  ASC  with  scenario 
development,  experimental  design,  and  batderoom  visualization  efforts  for  a  SIMAF 
capability  demonstration. 

This  thesis  uses  multivariate  analysis  and  visualization  tools  to  develop  an 
approach  for  reducing  the  dimensionality  of  multiple  campaign  level  measures  of 
effectiveness  for  a  notional  Analysis  of  Alternatives  (AoA)  study.  Additionally,  the  thesis 
advances  an  AoA  visualization  paradigm  for  the  SIMAF  capabihty  demonstration. 

The  results  of  this  study  suggest  that  multivariate  data  reduction  techniques  and 
user  interactive  visualization  of  multivariate  analysis  results  can  be  employed  to  combine 
multiple  MOEs  into  a  reduced  set  of  interpretable  factors  capturing  the  operational 
effectiveness  performance  of  competing  acquisition  alternatives.  The  thesis  research  also 
successfully  demonstrated  a  visual  data  mining  approach  applied  to  the  visuahzation  of 
campaign  level  analysis  results  and  the  cost/effectiveness  integration  of  an  AoA  effort. 
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ANALYSIS  OF  ALTERNATIVES:  MULTIVARIATE  CONSIDERATIONS 


1.  Introduction 


1.1  Background 

Modeling  and  simulation  (M&S)  is  increasingly  chosen  as  a  tool  to  aid  in  defense 
acquisition  decisions.  The  M&S  Master  Plan,  DoD  5000.59-P,  describes  the  DoD  vision 
of  “synthetic  environments”  representing  every  potential  opponent  in  any  region  of  the 
world  with  realistic  interactions  for  research,  development,  and  test  and  evaluation 
activities  in  defense  acquisition  (DoD  5000.59-P,  1995:2-2).  These  “synthetic 
environments”  would  potentially  link  several  types  of  M&S  including  operations  with 
real  equipment  in  the  field  (live);  war  games,  models,  and  analytical  tools  (constructive); 
and  systems  and  troops  in  simulators  fighting  on  simulated  battlefields  (virtual)  (Schoen 
and  Starr,  1993:  845).  M&S  resources  augmenting  these  environments  would  be 
collocated  or  distributed  geographically  and  linked  through  high-speed  data  networks. 
The  1996  Quadrennial  Defense  Review  (QDR)  accentuated  the  need  for  improved 
analytical  tools  to  capture  the  key  variables  in  force-on-force  assessments  across  the 
spectrum  of  military  engagements,  from  minor  contingencies  to  major  regional  conflicts 
(Holzer,  1997:  28). 

The  Air  Force’s  Aeronautical  Systems  Center  (ASC)  is  currently  developing  a 
capability  to  link  models,  simulations,  hardware-in-the-loop,  operator-in-the-loop,  and 
system-in-the-loop  resources  to  create  a  robust  virtual  environment  that  will  support  the 
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assessment  of  alternative  systems  in  the  acquisition  process.  ASC’s  Simulation  & 
Analysis  Facility  (SIMAF)  will  use  the  Distributed  Interactive  Simulation  (DIS)  protocol 
to  link  extensive  resources  indigenous  to  Wright  Patterson  Air  Force  Base’s  Wright 
Laboratory.  Additionally,  DIS  will  facilitate  linkage  to  assets  remote  from  the  base 
providing  a  scaleable  virtual  environment  capability.  The  SIMAF  will  leverage  off  the 
extensive  analytical  expertise  organic  to  ASC  for  planning  and  post  processing.  The 
facility  will  function  both  as  a  virtual  integrator  of  models,  simulations,  and  hardware  via 
communications  and  networking  nodes  and  as  a  physical  gateway  for  ASC  modeling, 
simulation,  and  analysis  to  the  synthetic  battlespace  (Smith,  1997:  4). 

1.2  Problem  Statement 

The  program  management  office  for  the  SIMAF,  ASC/SM,  plans  a  capability 
demonstration  in  the  Spring  of  1998.  The  demonstration  will  showcase  the  capability 
SIMAF  provides  to  improve  analysis  underlying  the  Air  Force  acquisition  decision 
process.  The  event  will  be  attended  by  acquisition  decision  makers  -  senior  leaders  from 
the  Air  Force  Material  Command  (AFMC)  and  ASC,  and  prospective  SIMAF  users  - 
representatives  from  ASC’s  System  Project  Offices  (SPO)  and  from  the  Center’s 
analytical  teams.  AFIT/ENS  is  supporting  the  demonstration  with  scenario  development, 
experimental  design,  and  battleroom  visualization  efforts  (ASC,  1997:  1). 

1.3  Scope 

This  thesis  focuses  on  the  campaign  analysis  and  Analysis  of  Alternatives  (AoA) 
portion  of  the  SIMAF  tasking  including  the  data  visualization.  ASC/XRE  routinely 
conducts  and  oversees  modeling  and  simulation  efforts  to  support  AoA  efforts  (Logan, 
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1997).  The  AoA  study’s  comparison  of  the  relative  advantages  and  disadvantages  of 
possible  acquisition  alternatives  assists  the  decision  maker  in  selecting  the  alternative 
solution  providing  the  maximum  value  of  military  worth.  AoA  analyses  includes 
modeling  and  simulation  at  differing  analysis  levels,  few-on-few,  campaign,  etc.,  gauged 
to  appropriately  address  the  mission  effectiveness  requirements  of  the  potential 
acquisition. 

The  final  product  of  an  AoA  study  is  a  report  that  displays  a  comparison  of 
alternatives  in  terms  of  cost  and  effectiveness.  Literature  review  and  personal  interviews 
indicate  there  is  not  a  widely  accepted  approach  in  the  acquisition  community  to  combine 
multiple  measures  of  mission  effectiveness  into  a  single  effectiveness  measure  for  AoA 
reporting.  This  document  describes  and  demonstrates  one  strategy  to  address  the 
multivariate  nature  of  effectiveness  measures  via  multivariate  and  visualization  tools. 

ASC/SM  has  directed  an  end-to-end  scenario  for  the  capability  demonstration  that 
highlights  SIMAF’s  potential  to  integrate  data  flow  between  simulations  at  various  levels 
of  the  modeling  hierarchy  (engineering,  engagement,  mission,  and/or  cannpaign)  in 
virtual  and  constructive  environments  for  use  in  analysis  to  support  acquisition  decisions 
(Smith,  1997:  9).  High  fidelity  visualizations  of  the  simulations  and  analytical  results  at 
each  stage  of  the  scenario  are  required  to  present  demonstration  observers  with  a  coherent 
visual  rendering  of  SIMAF  operation  and  output  abilities.  The  visualizations  must  be 
robust  enough  to  support  the  information  requirements  of  both  viewers  with  limited 
formal  analytical  backgrounds  and  experienced  analysts. 

The  notional  acquisition  of  a  new  air  superiority  fighter,  the  F-XX,  was  chosen 
for  the  SIMAF  demonstration.  The  THUNDER  6.4.2  campaign  model  with  an 


3 


unclassified  Southwest  Asia  (SWA)  based  database  developed  by  ASC/XRE  was  selected 
for  the  simulation.  In  view  of  the  notional  nature  of  the  F-XX  and  the  fact  that  much  of 
the  data  on  the  actual  aircraft  and  equipment  provided  in  the  ASC  database  has  been 
notionalized  for  classification  purposes,  the  utility  of  the  output  of  the  simulation  is 
limited  to  qualitative  information  for  contrasting  acquisition  alternatives  rather  than 
generating  hard  quantitative  performance  measures. 

1.4  Thesis  Overview 

The  next  four  chapters  provide  a  detailed  description  of  the  thesis  effort.  Chapter 
two  summarizes  a  review  of  literature  published  on  topics  impacting  the  thesis  area  of 
interest  including  the  AoA  process,  military  M&S,  information  visualization,  design  of 
experiments,  and  multivariate  analysis.  The  specifics  of  the  methodology  applied  in  the 
thesis  are  discussed  in  chapter  three.  Chapter  four  provides  results  and  an  analysis  of  the 
data.  Finally,  conclusions  and  recommendations  for  areas  of  future  research  are 
presented  in  chapter  five. 

The  appendices  supplement  the  material  in  the  text.  Appendix  A  depicts  the 
suggested  format  of  an  AoA  Report  Format.  The  Air  and  Space  Power  Validation 
Group’s  (ASPVG)  listing  of  the  measures  THUNDER  can  assess  at  the  campaign  level  is 
at  Appendix  B.  Appendix  C  lists  the  output  metrics  and  their  abbreviations.  THUNDER 
data  files  modified  for  the  thesis  effort  are  documented  in  Appendix  D.  The  output 
responses  from  the  THUNDER  runs,  calculations  used  in  the  analysis  of  the  results, 
intermediate  data  sets  and  schema  used  for  the  visualization  data  set  are  included  in  the 
remaining  appendices. 
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2.  Literature  Review 


2.1  Analysis  of  Alternatives  (AoA) 

An  AoA  is  a  detailed  analysis  of  the  advantages  and  disadvantages  of  the 
potential  alternative  solutions  to  address  an  established  mission  need  (Draft  OASP  97-1, 
1997:6).  The  DoD  regulation  outlining  the  mandatory  procedures  for  Major  Defense 
Acquisition  Programs  (MDAP)  and  Major  Information  System  Acquisition  Programs 
(MAISAP),  DoDR  5000.2-R,  requires  preparation  of  an  AoA  for  aU  Acquisition 
Category  (ACAT)  I  and  lA  programs  (DoDR  5000.2-R,  1996:3).  ACAT I  and  lA  are 
MDAP  and  MAISAP  programs,  respectively.  Selection  criteria  for  the  ACAT 
designation  of  a  program  is  listed  in  several  documents  including  DODR  5000.2-R.  AoA 
study  efforts  are  performed  on  other  ACAT  programs  as  required. 

AoAs  provide  analytical  justification  for  selected  courses  of  action  (Diaz,  1992: 
79).  Alternative  solutions  considered  in  an  AoA  must  be  comprehensive  to  include 
current  systems,  modifications  to  current  systems,  commercial  off-the-shelf/government 
off-the-shelf  (COTS/GOTS)  possibilities,  systems  in  development,  non-developmental 
systems,  conceptual  systems  (if  they  can  be  fielded  within  the  time  constraints  imposed 
by  the  requirements),  and  systems  of  other  services  and  allies  (Draft  OASP  97-1, 
1997:16). 

The  AoA  process  results  in  a  report  discussing  the  analytical  rationale  for  the 
selection  of  the  best  solution  in  terms  of  cost  and  operational  effectiveness  to  support  a 
program  decision.  An  outline  of  the  suggested  format  for  the  AoA  report  is  attached  at 
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Appendix  A.  This  document  facilitates  the  acquisition  process  by  providing  a  linkage 
between  system  requirements  and  specific  measures  of  operational  effectiveness  (AoA 
Course,  1997: 10).  It  provides  an  audit  trail  that  weaves  all  procurement  justifications 
together  (Diaz,  1992:  81).  The  analysis  helps  the  decision  maker  judge  whether  or  not 
any  of  the  proposed  alternatives  to  an  existing  system  offer  sufficient  military  and/or 
economic  advantage  to  be  worth  the  cost.  In  addition  to  providing  a  quantitative 
assessment  of  each  alternative,  the  AoA  should  assess  sensitivities  of  each  potential 
solution  to  uncertainty  in  key  assumptions  (e.g.  threat)  and/or  system  variables  (e.g.  user 
specified  performance  capabilities  -  airspeed,  range,  payload,  etc.).  Guidelines  provided 
by  AFMC’s  Office  of  Aerospace  Studies  (OAS),  the  Air  Force’s  AoA  center  of  expertise, 
and  completed  studies  assist  the  study  team  in  operational  effectiveness  and  cost  analysis 
during  the  AoA  effort. 

2.1.1  Operational  Effectiveness  Methodology.  Combat  effectiveness  analysis  is 
a  measure  of  an  alternative’s  ability  to  meet  established  mission  requirements  in  an 
operational  environment  (Diaz,  1992:  87).  The  assessment  of  operational  effectiveness 
via  M&S  should  include  the  use  of  existing,  validated  models  to  evaluate  the  quantitative 
impact  on  mission  accomplishment  of  competing  system  alternatives  (Draft  OASP  97-1, 
1997:  19).  The  ability  of  the  solutions  to  nKet  mission  requirements  is  delineated  in 
several  ways.  High  level  mission  tasks  (MT)  are  determined  that  describe  the  tasks  a 
system  will  be  expected  to  perform.  Alternative  system  performance  is  then  measured  by 
the  degree  the  specified  tasks  are  accomplished.  Task  performance  is  gauged  by 
Measures  of  Effectiveness  (MOEs),  qualitative  measures  of  a  system’s  performance  or  a 
characteristic  that  indicates  the  degree  to  which  it  performs  a  task  or  meets  a  requirement 
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under  specified  conditions.  Subordinate  to  MOEs,  Measures  of  Performance  (MOP) 
provide  the  lowest  quantitative  measure  of  the  systems’  physical  performance  -  range, 
velocity,  etc.,  or  physical  characteristic  -  height,  weight,  volume,  etc.  (AoA  course. 


Determine 
Mission  Needs 

Determine 

Issues 

AoA  Study  Operational  Effectiveness  Analysis 


— lain — 

- TcTir!? - 

- - 1 

ALTl 

Determine 
Approach  to 
Evaluate 
MOEs 


Figure  1,  General  Approach  to  Effectiveness  Analysis  (AoA  Executive  Brief,  1997:21) 


1997:58).  The  general  approach  to  an  effectiveness  analysis  is  depicted  in  Figure  1. 

An  example  of  an  air  superiority  mission  task  might  be  the  number  of  days  to 
achieve  air  superiority.  The  solutions  proposed  in  an  AoA  are  assessed  on  how  well  they 
satisfy  the  mission  tasks.  Once  alternative  solutions  have  been  scoped,  the  next  step  is  to 
identify  MOEs  and  MOPs  to  evaluate  the  ability  of  the  solutions  to  support  the  mission 
tasks.  MOEs  should  be  chosen  that  directly  relate  to  a  systems  mission  tasks  and  overall 
mission  accomplishment.  These  measures  should  be  selected  to  reflect  the  contribution 
of  a  particular  system  to  the  outcome  of  battle,  not  just  how  far  it  can  shoot  or  how  fast  it 
can  fly  (Draft  OASP  97-1, 1997:23).  The  cost/effectiveness  assessment  stage  of  an  AoA 
uses  a  single  effectiveness  measure  for  each  alternative  solution. 
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2.1.2  Combining  Effectiveness  Measures.  While  there  are  processes  to  quantify 
MOEs,  there  is  not  a  universally  accepted  method  to  combine  several  MOEs  into  a  single 
effectiveness  measure  (Pinker,  Samuel,  and  Batcher,  1995:8).  Past  AoA  study  efforts 
have  predominantly  taken  two  approaches  to  deriving  a  single  overarching  effectiveness 
measxu'e  for  alternate  solutions.  One  approach  is  to  derive  a  single  measure  from  the  user 
at  the  outset  of  the  study  that  encapsulates  the  operational  effectiveness  of  the  system, 
e.g.  number  or  percentage  of  targets  killed.  Another  method  used  for  arriving  at  a  single 
effectiveness  measure  has  been  to  evaluate  several  MOEs  in  the  course  of  the  AoA  study, 
then  selecting  one  of  the  measures  as  best  representing  the  system’s  ability  to  accomplish 
its  mission  for  use  in  the  cost/effectiveness  analysis. 

Strategies  have  been  proposed  to  combine  measures  of  effectiveness  into  a  single 
measure.  One  method  proposed  for  combining  MOEs  is  to  use  a  linear  combination  of 
the  various  measures  of  crucial  importance  to  the  decision  maker  to  form  a  single  index 
for  each  alternative  (Pinker,  Samuel,  and  Batcher,  1995:9).  The  measures  are  weighted 
to  reflect  their  criticality  to  the  decision  maker.  The  difference  in  units  between  the 
factors  in  this  approach  is  addressed  by  normalizing  the  data  to  a  baseline  alternative.  A 
shortcoming  of  this  method  is  that  the  choice  of  the  baseline  for  normalization  could 
possibly  change  the  rankings.  Another  strategy  to  reduce  MOE  dimensionality  proposes 
a  similar  approach,  but  normalizes  the  data  across  alternatives  using  the  best  value  as  the 
baseline  for  each  factor.  This  approach  to  normalizing  the  factors  ensures  a  consistent 
ranking  insensitive  to  the  addition  of  new  alternatives  (Melese  and  Bonsper,  1996:17). 

Air  Force  AoA  guidance  generally  discourages  combining  MOEs  into  a  single 
weighted  measure.  However,  the  guidance  acknowledges  that  weighting  schemes  can  be 
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useful  to  the  analysis  if  the  weighting  methodology  is  clearly  explained  to  facilitate  an 
accurate  interpretation  of  the  results  (Draft  OASP  97-1, 1997:77).  In  addition  to 
operational  effectiveness,  AoAs  also  include  cost  analysis. 

2.1.3  Cost  Analysis.  The  AoA  process  views  cost  as  an  independent  variable 
(CAIV).  CAIV  is  a  concept  emphasizing  cost  or  unit  price  as  a  constant.  Cost  and 
operational  effectiveness  are  considered  equals  necessitating  trade-offs  throughout  the 
acquisition  process.  An  affordable  price  for  a  system  is  established,  then  either 
performance  or  schedule  is  adjusted  to  meet  that  price  (Draft  OASP  97-1, 1997:18).  A 
Life  Cycle  Cost  (LCC)  estimate  is  completed  on  all  alternatives  in  the  AoA.  Costs 
included  in  the  LCC  are  development,  installation,  procurement,  operations  and  support 
(O&S),  and  disposal  costs  of  the  system.  The  final  stage  of  the  AoA  effort  is  the 
integration  of  the  cost  analysis  with  the  effectiveness  measures. 

2.1.4  Cost/Effectiveness  Analysis.  There  are  several  approaches  to  cost 
effectiveness  analysis  of  alternatives;  1)  a  direct  comparison  of  cost  versus  effectiveness, 
2)  a  comparison  on  equal  cost  basis,  and  3)  a  comparison  on  an  equal  effectiveness  basis. 
A  direct  comparison  fuses  together  the  equal  cost  basis  and  effectiveness  approaches  by 
the  addition  of  a  cost  ceiling  provided  by  the  SPO  and  an  effectiveness  ceiling  provided 
by  the  user  to  filter  out  unacceptable  alternatives  (Draft  OASP  97-1, 1997: 92).  A 
comparison  on  an  equal  cost  basis  would  be  how  many  targets  could  be  neutralized  for 
XX  dollars  with  the  use  of  each  alternative  in  a  scenario.  An  equal  effectiveness  basis 
might  be  the  cost  of  each  alternative  to  neutralize  XX  targets.  Figure  2  depicts  an  AoA 
cost/effectiveness  comparison.  A  graph  of  this  type  is  typically  the  summary  of  an  AoA 
effort.  The  figure  would  seem  to  clearly  indicate  alternative  7  as  the  preferred  choice. 
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Figure  2,  Cost/Effectiveness  Comparison  (Draft  OASP  97-1, 1997:92) 
However,  other  factors  come  into  play  in  the  determination  of  the  best  solution  that  are 
not  considered  in  the  AoA,  -  politics,  force  structure,  schedule,  etc. 


2.2  Military  M&S 

Models  are  mathematical  representations  of  a  real-world  system.  A  simulation  is 

the  operation  of  a  real-world  system  or  process  over  time.  The  behavior  of  a  system  as  it 

evolves  over  time  is  studied  by  developing  a  simulation  model  (Banks,  Carson,  and 

Nelson,  1996:3).  Dimensions  of  DoD  M&S  are  depicted  in  Figure  3.  As  depicted  in  the 

figure,  military  M&S  employed  in  training,  acquisition,  and  analysis  includes  subsystem 

(engineering  level)  through  campaign/  theater  level  types. 

Models  for  defense  analysis  range  from  engineering  models  of  specific  systems 
(an  airborne  radar  system  for  example)  to  engagement  models  (surface-to-air 
(SAM)  systems  engaging  aircraft)  to  mission  models  (a  model  of  a  flight  of 
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Figure  3,  DoD  M&S  Dimensions  (DoD  5000.59-P,  1995:2- 

aircraft  from  takeoff  to  target  engagement  and  return)  to  the  campaign  model 
representing  a  set  of  missions,  operations,  or  battles  in  a  military  campaign 
(Hillestad,  Bennett,  and  Moore,  1996:4). 

Classes  of  modeling  and  simulation  cover  the  spectrum  from  live  or  operator-in-the-loop 
M&S,  to  virtual  M&S  encompassing  a  confederation  of  live,  hardware/operator-in- the- 
loop,  and  analytical  resources  operating  in  real-time,  to  constructive  M&S  composed  of 
analytical  resources. 

2.2.1  Campaign  Modeling.  Campaign  modeling  is  characterized  as  the  highest 
level  in  the  DoD  modeling  hierarchy  as  depicted  in  Figure  4.  As  the  figure  depicts, 
resolution  is  lowest  and  aggregation  is  highest  at  the  top  of  the  modeling  hierarchy.  The 
subsysteip/component  model  at  the  bottom  of  the  pyramid  must  render  the  system  at  a 
high  degree  of  granularity  for  engineering  level  analysis.  The  level  of  detail  within  a 
particular  system  must  be  aggregated  to  included  many  systems  within  a  campaign  level 
simulation  and  still  run  at  a  reasonable  speed. 
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An  aggregated  combat  model  groups  individual  combatants  into  larger  ‘units’, 
typically  using  the  real  world  hierarchical  command  organizations  of  the  force  to 
determine  natural  groupings  in  the  model.  Thus  the  entities  in  an  aggregated 
model  might  be  company,  battalion,  or  division  size  units.  Aggregated  models  do 
not  contain  detailed  information  about  individuals  making  up  a  unit  or  about 
individual  engagements  making  up  a  battle.  Thus  they  can  model  larger  forces  in 
theater  and  campaign  scenarios  (Hartman,  1996:1-3). 

A 


Campaign/Theater 


Figure  4,  DoD  M&S  Hierarchy  (THUNDER  Analyst  Manual,  1995:3) 

The  source  of  much  of  the  data  to  populate  a  campaign  model  is  computations  of 
other,  higher-resolution  models,  e.g.  outputs  of  mission,  engagement,  and  engineering 
level  models.  The  process  of  calibrating  a  campaign  model  with  data  from  a  higher 
resolution  model  requires  a  knowledgeable  analyst  to  determine  how  to  aggregate  the 
data,  when  the  approximation  is  good  enough,  what  cases  to  use,  and  what  parameters  to 
adjust  to  achieve  a  good  approximation  (HUlestad,  Bennett,  and  Moore,  1996:  16). 

Analysis  of  objectives  is  the  best  approach  to  understanding  military  campaigns. 
Objectives  guide  decisions  at  every  level  of  DoD  from  the  National  Command  Authority 
to  junior  officers  engaged  in  combat  (Piinie  and  Gardiner,  1996:3).  Modeling  output  is 
used  to  assess  Measures  of  Outcome  (MOO)  of  a  campaign.  MOOs  are  a  level  above  the 
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MTs  as  described  in  the  discussion  of  operational  effectiveness  methodology.  MOEs 
support  the  assessment  of  MTs.  MTs,  in  turn,  are  used  to  evaluate  the  accomplishment  of 
MOOs.  A  commander  achieves  operational  objectives,  the  MOO,  within  the  concept  of 
operations  or  the  campaign  plan  to  accomplish  his  mission,  e.g.  dominate  opposing 
operations  in  the  air  (Prinie  and  Gardiner,  1996:16).  Analysis  at  the  campaign  level  is 
concerned  with  the  cumulative  long  term  effects  of  kills  and  losses  on  the  outcome  of 
theater  level  conflict  of  campaign  duration.  Determination  of  the  relative  values  of  all 
targets  killed  and  losses  is  the  exclusive  domain  of  Theater/Campaign  Level  analysis 
(ASC/XR  Levels  of  Analysis,  1997:3). 

Analysis  of  Alternatives  (AoA)  generally  involve  campaign  level  modeling  to 
support  the  assessment  of  potential  solutions  (Starr,  1997:254).  Campaign  level 
modeling  is  too  coarse  grained  to  be  used  to  for  determining  marginal  value  or  the  impact 
of  small  force  increments  (AFMCP  800-66, 1993:17).  The  value  of  campaign  analysis  in 
military  applications  is  its  utility  in  capturing  the  interactions  among  the  total  forces 
involved:  air,  ground,  naval,  and  coalition  forces.  Campaign  models  are  often  referred  to 
as  theater  models  because  one  theater  of  operation  is  involved,  although  several 
campaigns  may  be  included  in  the  theater  (Hillstead,  Bennett,  and  Moore,  1996:4). 
Theater  level  models  involve  complex  interactions  among  many  different  players  and 
organizations  making  it  difficult  to  assess  the  effect  a  single  variable  (e.g.  introduction  of 
a  new  fighter  aircraft)  has  on  the  outcome  of  the  conflict.  The  challenge  of  the  military 
analyst  is  to  develop  modeling  techniques  to  use  high  resolution  insights  within  a 
campaign  (Friel,  1992: 129).  The  appropriate  choice  of  performance  measures  is  pivotal 
to  relevant  campaign  insights.  A  campaign  analysis  is  the  analytical  equivalent  of  a 
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military  campaign  executed  under  an  operations  plan  that  defines  a  series  of  operations 
by  integrated  forces  (Friel,  1992:130). 

2.2.2  THUNDER.  THUNDER  is  a  data  driven,  force-on-force  theater  level 
model.  The  data  files  (80+)  in  THUNDER  create  the  simulation  scenario  by  defining, 
terrain,  forces,  equipment,  and  weapon  systems.  The  model  stochastically  simulates  the 
air  war  providing  outputs  for  theater  analyses  supporting  force  structure  evaluations, 
tactics  development,  war-gaming,  and  analysis  of  alternatives  efforts  (THUNDER 
Analyst  Manual,  1995:1). 

M&S  credibility  is  measured  by  verification  and  validation  and  formally  approved 
as  adequate  for  use  in  a  particular  application  by  accreditation  (VV&A)  (DMSO,  1996:1- 
3).  THUNDER  is  a  legacy  simulation. 

A  good  legacy  simulation  is  characterized  by  a  long  history  of  consistent  use  and 
development  by  an  active  (usually  large)  user  group,  good  configuration 
management  and  documentation,  and  widely  recognized  community  acceptance 
of  its  results  (DMSO,  1996:1-7). 

The  user  is  responsible  for  the  VV&A  of  legacy  M&S.  Thorough  and  well  maintained 
documentation  makes  verification  fairly  straightforward.  THUNDER  has  not  received  a 
formal  results  validation  using  the  thesis  scenario  data  files  to  compare  simulation  output 
with  the  actual  performance/  employment  of  the  aircraft  represented.  This  validation 
would  be  accomplished  by  the  user.  The  user  can  also  perform  a  conceptual  model 
validation  by  comparing  THUNDER’S  assumptions,  limitations,  and  design  elements  to 
their  specific  requirements.  After  completing  the  V&V,  the  accreditation  agent,  the 
legacy  M&S  user,  formally  accredits  that  a  specific  simulation  can  be  used  for  a  specific 
application,  based  on  objective  evidence  of  suitability  for  the  application  (DMSO, 
1996:1-9). 
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Air  Force  AoA  guidance  underscores  the  use  of  existing,  validated  models  to 
evaluate  the  quantitative  impact  competing  system  solutions  have  on  mission 
accomplishment  (Draft  OASP  97-1,  1997:19).  The  prohibitive  cost  and  time  to  develop  a 
large  scale  model  is  a  strong  argument  for  the  use  of  existing,  widely  accepted  models. 

In  addition  to  analyses  supporting  defense  acquisitions  and  force  structure 
assessments,  THUNDER  has  been  used  as  a  campaign  level  analysis  tool  in  several 
recent  theses.  Grier  developed  a  THUNDER  based  quick  turn  evaluation  tool  that  links 
cost  and  capabilities  of  alternative  force  structures  (Grier,  1996:4).  A  Response  Surface 
Methodology  (RSM)  based  approach  to  develop  a  tool  for  force  structure  assessments 
utilizing  THUNDER  output  was  advanced  by  Farmer  in  his  thesis  work  (Farmer,  1996:6- 
1).  Forsythe’s  thesis  focused  on  THUNDER’S  air  apportionment  process.  He  employed 
an  RSM  technique  to  provide  insights  on  aircraft  apportionment  and  campaign  outcome 
relationships  that  facilitate  the  evaluation  of  non-material  solutions  in  acquisition 
decisions  (Forsythe,  1994:1-1).  Webb  performed  a  sensitivity  analysis  on  selected 
THUNDER  5.9  outputs  to  inputs  specified  by  ASC/XR  (Webb,  1994:1.6). 

2.3  Information  Visualization 

The  purpose  of  information  visualization  is  to  assist  the  analyst  and  the 
decisionmaker  through  a  visual  rendering  of  analytical  results.  Although  closely  related 
to  scientific  visualization,  information  visualization  provides  a  geometric  structure  to 
abstract,  symbolic,  and  numeric  information  (Talbert,  1997:21).  Scientific  visualization 
is  mainly  employed  to  visually  present  the  numerical  output  of  a  simulation,  e.g.  the 
simulated  airflow  around  an  aircraft  (Edwards,  1992:1). 
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Leveraging  on  a  human’s  natural  ability  to  recognize  patterns  and  structures  of 
images  versus  tables  of  numbers,  a  properly  implemented  information  visualization 
allows  the  presentation  of  large  volumes  of  data  in  a  format  that  can  be  easily  assimilated 
by  the  viewer  (Wright,  1995:19).  However,  it  is  incumbent  for  the  analyst  to  format  the 
visualization  into  the  context  of  the  decisionmaker  (Jones,  1996:29).  The  visualization 
should  adhere  to  basic  principles  of  graphic  design  to  achieve  the  maximum  impact  on 
the  viewer.  Sound  graphic  design  dictates  that  displays  of  data  focus  on  structure, 
maximizing  the  ink  (pixel)  to  data  ratio  for  high  data  densities  (e.g.  representing  many 
numbers  in  a  small  area).  Additionally,  the  data  should  be  presented  in  various  levels  of 
granularity,  from  aggregate  to  fine  resolution  (Tufte,  1983:14).  The  image  presented 
must  enable  users  to  quickly  extract  the  information  they  need,  understand  its  import,  and 
make  decisions  (Gershon,  Eick,  1997:29). 

A  non- interactive  example  of  information  visualization  used  in  a  military 
simulation  application  is  the  viewer.  The  viewer  allows  the  user  insight  into  the 
simulation  by  displaying  the  values  of  simulation  parameters  or  representations  of  the 
simulation  and  any  connected  databases  (Molitoris  and  Taylor,  1995:1 173).  A  stealth 
viewer  provides  a  non-interactive  view  of  the  air  war  from  various  perspectives  (cockpit, 
ground,  “God’s  eye”)  and  temporal  (live  or  recorded)  regimes  (Zyda,  et.  al,  1993:251). 
Stealth  viewing  gives  the  opportunity  for  the  analyst  and  decision  maker  to  better 
evaluate  the  performance  of  an  aircraft  model  by  viewing  the  visual  rendering  of  the 
simulation  from  different  aspects. 

Information  visualizations  provided  via  graphics  such  as  scatterplots,  histograms, 
and  3D  spin  plots  can  reveal  the  structure  of  data  overlooked  by  the  application  of 
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automated,  pattern  detected  algorithms  (Elder  and  Pregibon,  1997:103).  Two 
visualization  techniques  that  assist  both  the  decisionmaker’s  understanding  and  the 
researcher’s  analysis  of  the  large  data  sets  are  animation  and  data  mining. 

2.3.1  Animation.  Interactive  visualization  environments  have  the  potential  to 
provide  practical  solutions  to  real  world  problems  involving  complex  data  more  rapidly 
than  either  a  human  or  computer  operating  independently  (Uthurusamy,  1996:564). 
Animation  promotes  a  greater  understanding  of  the  information  visualization  by  allowing 
the  user  to  interact  with  a  scene  by  rotating  the  graphic  to  view  the  image  from  different 
angles,  moving  through  the  scene  to  zoom  into  an  area  of  interest,  and  selectively 
choosing  higher/lower  detail  of  specific  objects  to  filter  the  data  and  potentially  reveal 
patterns/anomalies  in  the  numbers  displayed. 

Often  it  is  difficult  to  target  a  single  point  or  a  bounded  subset  of  points  of 
statistical  interest  to  an  analysis  out  of  a  large  data  set.  Interactive  visualization  allows 
the  analyst  to  view  numerous  predictor/response  combinations  revealing  a  wide  range  of 
patterns  without  having'  to  choose  these  pattern  parameters  as  goals  in  advance  (Elder  and 
Pregibon,  1997:103). 

2.3.2  Data  Mining.  Data  Mining  via  visualization  allows  the  identification  and 
cataloguing  of  trends  in  large  databases  by  applying  pattern  recognition,  statistical,  and 
mathematical  techniques  (Berry,  1997:96).  Data  mining  specific  statistical  packages  are 
weighted  more  heavily  in  analytical  tools  to  address  non-linearity,  outliers,  and  non- 
numerical  data  than  many  high-end  statistical  packages  (Pass,  1997:26).  The  goals  of 
data  mining  are  descriptive  and  prescriptive.  Predictive  from  the  aspect  of  forecasting 
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future  outputs.  Descriptive  from  the  aspect  of  plumbing  understandable  patterns  that 
describe  the  data. 

Selecting  the  appropriate  data  mining  technique(s)  for  a  data  set  involves  two 
steps;  1)  translate  the  problem  at  hand  into  a  series  of  data  mining  tasks  and  2)  understand 
the  data  in  terms  of  the  fields,  contents,  and  structure  of  relationships  between  the  records 
(Berry,  1997:413).  Data  mining  techniques  (algorithms)  target  six  high  level  tasks  to 
achieve  these  goals  -  classification,  regression,  clustering,  summarization,  dependency 
modeling,  or  change  and  deviation  detection  (Fayyad,  Pietetsky-Shapiro,  and  Smyth, 

1996;  13).  Classification  bins  data  into  predefined  classes.  Regression  pairs  data  items 
with  prediction  variables.  Clustering  segments  a  diverse  population  into  subsets/clusters 
of  more  homogeneous  subgroups  (Berry,  1997:55).  Summarization  characterizes  the  data 
via  a  compact  description,  e.g.  mean  and  standard  deviation.  Dependency  modeling 
develops  a  model  describing  the  dependencies  between  variables  (Fayyad,  Pietetsky- 
Shapiro,  and  Smyth,  1996:15).  Change  variation  and  deviation  detection  focuses  on 
significant  changes  in  the  data  from  previously  measured  values. 

The  algorithms  for  data  mining  are  proliferate,  but  can  generally  can  be  described 
in  terms  of  three  unifying  characteristics;  model  representation,  model  evaluation,  and 
search  (Fayyad,  Pietetsky-Shapiro,  and  Smyth,  1996:16).  Model  representation  describes 
the  patterns  found  in  the  data.  Model  evaluation  is  validation  of  the  model  with  actual 
data.  The  search  characteristic  is  self  explanatory.  Users  can  interact  with  aggregate  data 
while  having  the  capability  to  drill-down  to  perform  a  detailed  analysis  of  the 
performance  of  a  specific  aircraft  or  weapons  type  over  the  campaign  (Wright,  1997:68). 
Detailed  analysis  via  data  mining  helps  identify  patterns/anomalies  for  the 
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analyst/decisionmaker.  It  augments  visualization  of  the  data  by  providing  a  focus  and 
preventing  the  data  from  overloading  the  viewer.  Knowledge  discovery  from  databases 
(KDD)  takes  the  process  a  step  further  by  using  personal  expertise  and  interpretative 
skills  to  derive  useful  knowledge  from  the  data  (Fayyad,  Pietetsky-Shapiro,  and  Smyth, 
1996:4).  Figure  5  encapsulates  the  KDD  process. 


Figure  5,  Overview  of  Knowledge  Discovery  from  Databases  Process  (Fayyad,  Piatetsky- 
Shapiro,  and  Smyth,  1996: 10) 

2.4  Design  of  Experiment  (DOE) 

M&S  to  support  acquisition  should  include  experimental  design  to  facilitate  the 
investigation  of  how  sensitive  the  results  of  the  analysis  are  to  changes  in  the  input 
parameters  (Starr,  1997:253).  DOE  provides  a  planning  tool  for  determining  the 
configurations  of  the  input  parameters  to  simulate  that  will  provide  the  most  information. 
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The  efficiency  of  carefully  designed  experiments  is  much  higher  than  an  arbitrary 
sequence  of  runs  to  see  what  happens  (Law  and  Kelton,  1991:657). 

Input  parameters  to  the  experiment  such  as  the  decision  variables,  the  structural 
assumptions,  and  the  parameters  of  the  random  variables,  are  called  factors  (Banks, 
Carson,  and  Nelson,  1996,  500).  The  simulation  is  run  at  various  values,  or  levels  of  the 
factors  to  provide  output  performance  responses  over  the  desired  range  of  interest.  A 
combination  of  factors  at  a  specified  level  is  called  a  treatment  or  a  design  point.  The 
experimental  design  is  the  collection  of  design  points  to  be  investigated  in  an  experiment. 
Experimental  responses  are  used  by  the  researcher  in  determining  if  there  are  any 
differences  between  the  levels  of  the  factors. 

The  factors  can  be  either  quantitative  or  qualitative.  A  quantitative  factor  is  one 
whose  levels  can  be  associated  with  a  numerical  scale,  e.g.  temperature,  pressure,  or  time. 
Qualitative  factors  are  factors  whose  levels  cannot  be  arranged  in  order  of  magnitude,  e.g. 
batches  of  raw  material,  work  shifts,  etc.  (Montgomery,  1976:55).  The  levels  of  the 
factors  can  be  specifically  chosen  or  picked  at  random  from  all  possible  factor  levels. 

Randomness  and  replication  are  applied  to  derive  realism  and  statistical 
inferences,  respectively,  from  simulation  output.  Randomness  of  events  in  simulation 
imitates  real  life  to  portray  uncertainty  (Banks,  Carson,  and  Nelson,  1996:25). 
Replications  are  repetitions  of  the  experiments  at  a  specific  design  point. 

2.4.1  Replications.  Simulations  are  classified  as  either  terminating  or  non¬ 
terminating.  Terminating  simulations  run  for  a  specific  duration,  whereas  non¬ 
terminating/steady  state  simulations  continue  for  a  very  long  time.  The  determination  of 
the  simulation  type  depends  on  the  study  objectives  and  the  nature  of  the  system  (Banks, 
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Carson,  and  Nelson,  1996:436).  The  experimental  design  for  terminating  simulations 
calls  for  multiple  replications  with  the  length  of  each  replication  determined  by 
prespecified  initial  and  final  conditions  (Nelson,  1992:127). 

The  number  of  replications  can  be  determined  to  give  a  user  specified  precision 
within  a  confidence  interval  by  first  making  an  initial  sample  size  of  replications.  Four 
to  five  replications  is  recommended  for  n,,  (Banks,  Carson,  and  Nelson,  1996:449).  The 
number  of  replications  can  be  determined  to  provide  an  estimate  of  the  output  mean,  IJ. , 
to  a  user  specified  degree  of  precision  by  a  sequential  procedure,  adding  new  replications 
one  at  a  time  (Law  and  Kelton,  1991:538).  The  procedure  does  not  require  normality  of 
the  random  variables,  only  that  the  variables  are  independent  and  identically  distributed 
(i.i.d.).  This  method  ensures  only  the  number  of  replications  are  accomplished  that  need 
be  in  order  to  achieve  a  prespecified  absolute  error  level.  To  ensure  the  half-length(h.l.) 
of  the  100(1- a )%  confidence  interval  is  met,  a  sample  of  n  must  be  chosen  such  that 
n  >  Hq  and 


.  .  ta/2.n-lSo  ^ 

hd.  = - m —  ^ ' 

nr 


(2.1) 


where  t  is  the  100*(l-a/2)  percentage  point  of  the  t  distribution  with  n-1  degrees  of 
freedom,  Sq  is  an  initial  estimate  of  the  population  standard  deviation,  n  is  the  sample 
size,  ande  is  the  user  defined  error  criterion.  Solving  equation  (2.1)  gives  n  is  the 
smallest  integer  satisfying  n  >  and 


ft. 


n> 


a/2,n-l^0 


(2.2) 


Since  tn,y2.„-i  ^  Za/2  ’  ^  Initial  estimate  for  is  n 
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where  z  is  the  100*(l-a/2)  percentage  point  of  the  normal  distribution  (Banks,  Carson, 
and  Nelson,  1996:448). 

2.4.2  Randomness.  Randomness  in  a  simulation  experiment  is  achieved  by  using 
pseudorandom  numbers  that  are  controlled  via  a  random  number  seed  or  stream.  The 
assignment  of  a  different  random  seed  or  stream  to  the  start  of  each  replication  is 
normally  not  necessary  since  most  simulation  languages  begin  subsequent  replications 
using  random  numbers  from  where  the  previous  rephcation  finished  (Nelson,  1992:128). 
This  is  the  case  with  the  random  number  allocation  of  THUNDER.  The  simulation’s 
SIMSCRIPT  n.5  language  employs  a  random  number  generator  with  a  period  of 
approximately  ten  billion  numbers  (THUNDER  Analyst  Manual,  1995:96). 

2.5  Multivariate  Analysis 

Multivariate  analysis  focuses  on  the  correlation/covariance  relationships  between 
three  or  more  variables  (Dillon  and  Goldstein,  1984:2).  Both  covariance  and  correlation 
describe  the  dependencies  between  variables.  The  variance-covariance  matrix,  C,  is 

formulated  as  follows: 


S  =  X''X-(l/n)(X’'l)(l''X) 

(2.4) 

C=a/(n-l))« 

(2.5) 

where  S  is  the  mean  corrected  sum  of  squares  and  cross  products  matrix,  X  is  the  data 
matrix,  n  is  the  number  of  observations,  and  1  is  an  (n  x  1)  vector  of  I’s.  It  is  difficult  to 


employ  covariance  as  an  absolute  measure  of  dependence  because  its  value  depends  on 


scale  of  measurement  (Wackerly,  Mendenhall,  and  Schaeffer,  1996:224).  The  correlation 
matrix,  R ,  is  used  when  there  are  differences  in  scale.  The  correlation  matrix  is  related 

to  covariance  as  depicted  in  its  formulation. 

/?  =  DSD  (2.6) 

D  is  a  diagonal  matrix  with  diagonal  elements  ^Sf'  where  j  is  variables  (columns)  of 
the  input  data . 

Multivariate  analysis  techniques  are  applied  to  facilitate  data  reduction,  sorting 
and  grouping,  investigating  dependence,  prediction,  or  hypothesis  testing.  The  choice  of 
the  techniques  is  based  on  the  objective  of  the  analysis.  Three  multivariate  techniques 
commonly  used  for  data  reduction  are  Principal  Component  Analysis,  Factor  Analysis, 
and  Cluster  Analysis. 

2,5.1  Principal  Component  Analysis  (PCA).  The  objective  of  PCA  is  to  reduce 
the  variable  data  dimensionality  to  a  set  of  linear  combinations  that  explains  as  much  of 
the  variability  of  the  original  data  as  possible.  Choice  of  the  use  of  the  variance- 
covariance  or  the  correlation  matrices  in  PCA  depends  on  the  homogeneity  of  the  data. 
The  correlation  matrix  is  used  in  conjunction  with  standardized  data  (adjust  each  variable 
for  its  variability)  when  the  data  has  different  units  and  scales  (Dillon  and  Goldstein, 
1984:38). 

The  principal  components  loadings  matrix  describes  how  the  variances  load  on 
the  variables.  The  matrix  is  extracted  from  either  the  variance-covariance  or  correlation 
matrix.  Extraction  of  the  loadings  from  the  variance-covariance  matrix  is  formulated  as 
follows: 
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(2.7) 


where  Ac  is  the  matrix  of  eigenvalues  for  C ,  and  A  is  a  diagonal  matrix  of  the  square 

roots  of  the  eigenvalues  of  C  (Bauer,  1997:50).  The  eigenvalues  of  C  form  the  top  row 

of  the  loadings  matrix.  The  correlation  matrix  using  standardized  data  is  the  basis  of 
loading  extraction  when  variables  have  grossly  different  variances  or  are  measured  in 
significantly  different  units.  The  formulation  of  the  extraction  of  loadings  from  R  is  the 

same  approach  as  for  the  C  loadings  extraction  depicted  in  (2.7)  above.  The  eigenvalues 

of  R  form  the  top  row  of  the  loadings  matrix. 

The  dimensionality  of  the  data  can  be  estimated  from  the  eigenvalues  extracted 
from  the  variance-covariance  matrix.  A  threshold  on  percentage  variance  (e.g.  85%)  can 
be  applied  to  the  variance-covariance  extracted  matrix.  The  number  of  successive 
components  required  to  extract  the  threshold  cumulative  variance  is  the  dimensionality  of 
the  data. 

Likewise,  dimensionality  of  correlation  matrix  extracted  data  can  be  estimated  in 
several  ways.  One  method,  attributed  to  Kaiser,  is  based  on  the  size  of  the  eigenvalues 
extracted  from  R .  Since  the  variance  each  standardized  variable  contributes  to  a 

principal  component  extraction  is  one,  components  with  eigenvalues  less  than  1  are  less 
important  from  a  variance  standpoint  than  the  observed  value  (Tabachnick  and  Fidell, 
1989:634).  Subsequently,  the  number  of  components  with  eigenvalues  >  1  is  the 
dimensionality  of  the  data.  A  graphical  approach  to  dimensionality  estimation  is  the 
scree  test.  The  test  requires  the  eigenvalues  to  be  plotted  in  sequential  order  of  their 
extraction.  The  number  of  eigenvalues  immediately  above  the  point  where  the  values 
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become  a  straight  line  plot  is  the  dimensionality  of  the  data  (Dillon  and  Goldstein, 
1984:48).  Dimensionality  assessment  is  one  reason  PCA  is  recommended  as  the  first 
step  in  Factor  Analysis.  PCA  reveals  a  great  deal  of  information  about  the  probable 
structure  and  nature  of  factors  (Tabachnick  and  Fidell,  1989:626). 

2.5.2  Factor  Analysis  (FA).  Where  PCA  emphasizes  the  differences  in  the 
variances  between  the  variables,  FA  targets  the  communalities  or  common  variability. 
PCA  has  the  most  utility  in  determining  a  small  set  of  linear  combinations  characterizing 
the  variance  in  the  data.  FA  best  describes  the  qualitative  and  quantitative  nature  of  the 
underlying  data  structure  (Dillon  and  Goldstein,  1984:55).  FA  is  applied  either  as  an 
exploratory  or  confirmatory  method.  The  exploratory  application  is  to  search  for  a 
common  structure  to  the  data.  The  confirmatory  utility  of  FA  is  to  test  a  hypothesis.  The 
factor  loading  matrix  is  used  to  interpret  the  variables.  The  factor  loading  matrix,  A ,  can 

be  determined  directly  by  the  formulation: 

A=VL^^  (2.8) 

where  V  are  the  eigenvectors  extracted  from  the  correlation  matrix  of  standardized  data 

and  L  is  a  diagonal  matrix  with  the  eigenvalues  of  R  on  the  diagonal.  A  factor  is 

interpreted  from  the  variables  that  are  highly  correlated  with  it. 

Extraction  techniques  do  not  generally  provide  an  interpretable  solution  without 
rotation  (Tabachnick  and  Fidell,  1989:623).  Rotation  is  usually  performed  after  factor 
extraction  to  maximize  high  correlations  and  minimize  low  ones.  One  problem  with 
extracted  FA  and  PCA  units  is  there  are  an  infinite  number  of  rotations  available,  all 
accounting  for  the  same  variance  but  with  factors  defined  slightly  differently.  The  final 
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choice  among  alternatives  depends  on  the  researcher’s  assessment  of  its  interpretabihty 
and  scientific  utility  (Tabachnick  and  Fidell,  1989:598). 

There  are  two  methods  of  factor  rotation;  orthogonal  and  oblique.  The  distinction 
between  the  two  methods  is  that  orthogonal  preserves  the  original  orientation  between 
factors  so  they  are  still  perpendicular  after  rotation.  Factor  orientation  is  not  preserved  in 
obhque  rotation  (Dillon  and  Goldstein,  1984:55).  Varimax  is  a  orthogonal  rotation 
technique  that  employs  maximization  of  variances  to  accentuate  high  factor  loadings  and 
reduce  low  ones. 

A  procedure  to  verify  that  the  appropriate  number  of  factors  have  been  extracted 
is  via  an  assessment  of  the  difference  between  the  correlation  matrix  produced  by 
observed  variables  and  the  correlation  matrix  from  factors  (reproduced  nrntrix); 

R  =R-R  (2.9) 

^  res  " 

where  R  is  the  matrix  of  differences  called  the  residual  correlation  matrix,  R  is  the 

~  RES 

original  correlation  matrix  with  communalities  on  the  diagonal,  and  R  is  the  correlation 

matrix  of  factors.  In  a  good  FA,  correlations  in  the  residual  matrix  are  small  indicating  a 
good  fit  between  observed  and  reproduced  matrices  (Tabachnick  &  Fidell,  1989:599). 

Factor  scores  give  the  projection  of  an  observation  on  the  common  factors,  in 
other  words,  its  location  in  common  factor  space.  This  information  can  provide 
additional  insight  into  the  structure  of  the  data  by  highlighting  patterns  of  common 
variation  (Dillon  and  Goldstein,  1984:96).  Factor  scores  cannot  be  calculated  directly  but 
rather  estimated.  One  approach  for  factor  score  determination  is  a  regression  type 
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technique.  First,  a  factor  score  coefficients  matrix,  B ,  is  determined  using  a  formulation 
similar  to  multiple  regression, 

B  =  R-^A  (2.10) 

where  R~^  is  the  inverse  matrix  of  correlations  between  factors  and  variables,  and  A  is 
the  factor  loadings  matrix.  The  factor  score  matrix,  F ,  is  determined  as  the  product 
between  the  standardized  data  matrix,  Z ,  and  the  factor  score  coefficient  matrix,  B , 
using  the  formulation; 

F=ZB  (2.11) 

Plots  of  factor  scores  facilitate  the  understanding  of  different  patterns  of  common 
variance  in  the  data  (Dillon  and  Goldstein,  1984:98). 

2.5.3  Cluster  Analysis.  Clustering  is  a  method  to  discover  structure  within  a 
complex  data  set  for  the  purpose  of  reducing  the  dimensionality  of  the  data.  In 
clustering,  data  or  variables  are  organized  into  relatively  distinct  clusters  each  with 
elements  having  a  high  degree  of  “natural  association”  (Anderberg,  1973:17).  The 
analysis  focuses  on  rows  or  individual  objects  of  the  data  matrix  reducing  the  number  of 
distinct  entities  by  grouping  them  into  clusters.  Grouping  is  accomplished  via  some 
similarity  measure  or  distance  measurement  such  as  the  Euclidean  distance  between 

respective  objects.  The  Euclidean  distance,  ||  ||,  between  two  vectors,  X  and  Y ,  is 

formulated  as  follows: 


nV2 


(2.12) 
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The  scale  invariance  of  Euclidean  distance  measurement  necessitates  the  input  data  be 
standardized  (each  variable  divided  by  its  standard  deviation)  (Dillon  &  Goldstein, 
1984:162).  The  hierarchical  clustering  technique  performs  successive  groupings  of  data 
or  divisions  of  data.  Additive  or  agglomerative  hierarchical  clustering  methods  continue 
until  all  the  data  is  grouped  into  a  single  cluster. 

Cluster  analysis  is  a  tool  for  suggestion  and  discovery  that  can  illuminate 
relationships  and  principles  previously  unnoticed.  The  results  of  the  analysis  may  be  a 
hypothesis  to  be  tested  or  be  compelling  enough  to  be  immediately  adopted  (Anderberg, 
1973:19). 
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3.  Methodology 


3.1  Introduction/Overview 

Chapter  three  includes  a  brief  discussion  of  the  overall  objectives  of  the  research, 
then  sequentially  recounts  the  analytical  methods  apphed  in  the  thesis  effort  to  reduce  the 
dimensionality  of  measures  of  effectiveness  for  AoA. 

3.2  Objectives 

The  goal  of  the  research  was  twofold,  1)  development  of  an  approach  to  reduce 
the  dimensionality  of  multiple  campaign  level  measures  of  effectiveness  for  an  AOA,  and 

2)  development  of  an  AoA  visualization  paradigm  for  the  capability  demonstration  of  the 
ASC  SIMAF.  The  two  objectives  overlapped  by  design. 

The  thesis  objectives  were  the  culmination  of  several  subobjectives: 

1)  Determine  an  AoA  scenario 

2)  Determine  measures  of  effectivness  and  THUNDER  output  to  support  the  analysis 

3)  Modify  THUNDER  data  files/report  output  to  support  analysis 

4)  Determine  a  design  of  experiments  for  the  simulation 

5)  Perform  the  simulation 

6)  Use  multivariate  and  visualization  tools  to  reduce  effectiveness  measure 
dimensionality 

7)  Use  visualization  tools  to  display  AoA  results 
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3.3  Scenario 


The  scenario  for  the  analysis  was  based  on  a  notional  established  mission  need  for 
a  new  air  superiority  fighter.  The  author  is  assuming  the  role  of  an  analyst  performing 
the  campaign  level  analysis  for  the  notional  using  command’s  AoA  study  director.  If  this 
was  an  actual  AoA  study  effort  for  an  air  superiority  fighter  acquisition,  the  study 
director  would  be  from  the  Air  Combat  Command  (ACC).  This  document  describes  the 
campaign  level  modeling  and  analysis  in  support  of  the  AOA  study .  The  results  of  the 
campaign  level  modeling  in  an  AOA  analysis  are  generally  documented  in  section  four  of 
the  AoA  final  report.  A  suggested  AoA  report  format  is  attached  at  Appendix  1. 
THUNDER  version  6.4.2  was  used  for  the  campaign  level  modeling.  The  three 
alternative  solutions  considered  in  the  AOA  study  include  the  current  system, 
modifications  to  the  current  system,  and  systems  in  development. 

The  Air  Force’s  current  air  superiority  fighter  and  the  baseline  in  the  experiment 
is  the  F-15C.  Acquisition  of  the  Air  Force’s  next  generation  air  superiority  fighter,  the  F- 
22,  was  not  considered  in  the  study.  The  modification  to  the  current  system  is  an 
improved  radar.  This  type  of  avionics  improvement  could  be  part  of  a  programmed 
upgrade  for  the  aircraft.  The  system  in  development  is  the  notional  F-XX.  The  three 
alternatives  were  examined  employing  two  air-to-air  armament  configurations:  1)  a 
standard  F-15C  load  including  four  AIM-9  missiles,  four  Advanced  Medium  Range  Air- 
to-Air  Missile  (AMRAAM)  missiles,  and  a  20  mm  cannon  and  2)  the  replacement  of  the 
four  AMRAAM  missiles  in  the  standard  load  with  a  new  radar  missile  variant,  the  AIM- 
X.  Subsequently,  the  resulting  test  matrix  includes  six  combinations  of  aircraft  and 
armament.  Experimental  design  is  described  in  detail  in  a  later  section. 
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The  mission  need  specified  that  the  new  fighter  perform  effectively  in  an  air 
superiority  role  when  employed  in  a  Southwest  Asia  (SWA)  scenario.  Hence,  the 
database  used  in  the  analysis  was  an  unclassified  SWA  THUNDER  scenario  provided  by 
ASC/XR.  The  campaign  scenario  was  a  modified  version  of  the  Middle  East  (ME) 
scenario  included  in  the  THUNDER  6.4.2  model  distribution.  Table  1  depicts  a  breakout 

Table  1,  Listing  of  Aircraft  With  Air  Superiority  Taskings  in  Scenario 


Side 

Type  Aircraft 

Number 

Blue 

F-15C 

124 

Blue 

F-14 

100 

Red 

MIG-29 

147 

Red 

MIG-21 

147 

of  Blue  and  Red  air-to-air  players  flying  air  superiority  as  their  primary  mission  in  the 
experiment  scenario.  The  Blue  air-to-air  players  and  their  beddown  locations  in  the 
databases  are  based  on  information  in  The  Conduct  of  the  Persian  War  -  Final  Report  to 
Congress  (Department  of  Defense,  1992:142).  The  source  for  the  information  on  Red 
air-to-air  players  is  the  book  Storm  Over  Iraq  (Hallion,  1992:146).  Red  air-to-air  players 
in  the  THUNDER  scenario  used  in  the  experiment  included  750  fighter  aircraft  that  were 
arbitrarily  positioned  among  20  deployment  bases.  The  F-15C  represented  55%  of  the 
Blue  aircraft  in  the  scenario  solely  tasked  with  an  air  superiority  mission. 
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3.4  THUNDER  Output 

The  output  selected  for  the  simulation  was  based  on  issues  measurable  at  the 

campaign  level.  The  overall  campaign  objective  for  the  notional  fighter  was  to  gain  and 

maintain  air  superiority.  The  Air  and  Space  Power  Validation  Group  (ASPVG)  assessed 

THUNDER  version  6.3  for  its  capability  to  measure  campaign  objectives.  The  output  of 

the  assessment  was  a  hsting  of  campaign  objectives  broken  down  into  two  subordinate 

levels;  operational  objectives  and  operational  tasks.  Additionally,  the  vahdation  group 

assigned  a  qualitative  measure  to  THUNDER  ability  to  assess  each  campaign  objective, 

operational  objective,  and  operational  task.  The  list  of  AS  VPG  operational  objectives 

THUNDER  measures  for  the  air  superiority  campaign  objective  are  as  follows; 

Defeat  Air  Attacks 

Suppress  Generation  of  Air  Sorties 

Suppress  Surface-Based  Air  Defenses 

Defeat  Attacking  Ballistic  Missiles 

Suppress  the  Generation  of  Ballistic  Missile  Launches 

(ASVPG,  1995:4) 

Operations  objectives  are  related  in  that  the  progress  toward  one  objective  assists  in 
attaining  another  objective  or  objectives  (Pirnie  and  Gardiner,  1996:20).  While  the  F-XX 
would  only  directly  impact  the  first  operational  objective  listed,  it  would  indirectly 
impact  the  remaining  issues  due  the  increase  of  strikers  reaching  their  targets  on  account 
of  a  more  effective  air  superiority  fighter.  Operation  tasks  are  tactical-level  objectives 
that  must  be  attained  to  accomplish  operational  objectives  (Pirnie  and  Gardiner, 

1996:19).  The  operation  tasks  for  each  of  the  operational  objectives  in  the  air  superiority 
campaign  objective  that  THUNDER  can  measure  are  listed  in  Appendix  B  (ASPVG, 
1995:4).  The  unclassified  THUNDER  data  base  can  measure  the  following  metrics  to 
assess  operational  tasks: 
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Red  aircraft  lost  due  to  Blue  air 

Total  number  of  Blue  aircraft  destroyed  on  the  ground 

Number  of  Red  aircraft  destroyed  in  the  open 

Number  of  Transportable  Erector/Launchers  (TEL)  killed 

Number  of  Acquisition  (ACQ)  radars  killed 

Number  of  Fire  Control  (FC)  radars  killed 

Number  of  Blue  Aircraft  lost  to  enemy  surface-to-air  (SAM) 

missile  threats  (Grier,  1996:  109) 

The  Blue  sorties  generated  for  the  30  day  war  changes  significantly  for  each  alternative 
solution.  Subsequently,  the  Blue  aircraft  lost  to  SAMs  was  measured  as  a  ratio  of  aircraft 
lost/sortie. 

Three  operational  task  metrics  were  added  to  those  listed  above;  Red  sortie 
generation  capability  on  day  30,  days  to  reach  air  supremacy,  and  Blue  air-to-air  losses/ 
sortie.  Red  sortie  generation  capability  on  day  30  is  the  ratio  of  Red  sorties  generated  on 
day  30  to  the  sorties  generated  on  day  1.  For  this  experiment,  days  to  air  supremacy  was 
measured  as  the  point  Red  sorties  generation  rate  decreased  to  10%  of  the  initial  total 
sortie  rate.  Air  Force  doctrine  characterizes  air  supremacy  as  the  absolute  control  of  air 
or  space  that  is  only  possible  when  the  enemy  does  not  possess  adequate  aerospace  forces 
capable  of  effective  interference  (AFM  1-1,  Vol  1, 1992:10).  For  example,  during  Desert 
Storm  the  United  States  commander  in  chief  of  the  Central  Command  (USCINCCENT) 
claimed  air  superiority  from  the  outset  of  the  conflict  but  waited  until  D-i-10  to  declare  air 
supremacy  when  he  judged  the  Iraq  air  forces  had  lost  the  capability  to  pose  a  serious 
threat  (Pimie  and  Gardiner,  1996:16).  The  Blue  air  losses/sortie  measure  of  effectiveness 
was  treasured  as  the  ratio  of  Blue  air-to-air  losses  to  the  total  Blue  sorties  for  the  30  day 
war.  An  overview  of  the  operational  task  n^trics  to  campaign  objective  hierarchy 
addressed  in  this  investigation  is  depicted  in  Figure  6.  Mappings  of  operational  task 
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metrics  to  operational  tasks  is  in  Appendix  B.  Appendix  C  lists  the  abbreviations  for  all 
output  variables  used  in  the  simulation. 


Campaign  Objective 

Gain  and  Maintain  Air  Superiority 

1 

Operational  Objectives 

Defeat 

Suppress 

Suppress 

Defeat 

Suppress 

Air  Attacks 

Generation 

Surface-Based 

Attacking 

Generation 

of 

Air  Sorties 

Air  Defenses 

Ballistic 

Missiles 

of  Ballistic 
Missile 

Launches 
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BLossSort 
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RedACQ 

at 

at 

RedFC 

unclassified 

unclassified 

level 

level 

Figure  6,  Campaign  Objective  to  Operational  Task  Metric  Hierarchy 


3.5  THUNDER  File  Modifications 

The  following  section  describes  the  modifications  to  THUNDER  data  files  in  the 
experiment.  THUNDER  file  names  are  in  italics.  Copies  of  the  modified  data  files  for 
the  experiment  are  depicted  in  the  order  discussed  in  the  text  at  Appendix  D. 

This  effort  was  not  a  sensitivity  analysis  or  trade-off  study  to  optimize  an  actual 
aircraft  design.  Quick-look,  single  repetition  runs  of  a  30  day  war  in  THUNDER  with 
various  RCSs,  missile  PkS,  radar  designs,  and  detection  capabilities  for  the  F-XX  and 
system  improvements  to  the  F-15C  were  performed.  The  exploratory  runs  were  made 
solely  to  implement  data  file  changes  that  provided  a  notional  aircraft/system 
modifications  with  a  significant  capability  improvement  to  attrit  red  air  at  a  level 
measurable  in  a  campaign  level  analysis.  The  control.dat  file  was  configured  to  run  a  30 
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day  war.  A  30  day  war  was  chosen  because  it  was  adequate  for  a  drawdown  of  Red 
sortie  generation  capability  to  assess  the  air  supremacy  output  discussed  in  section  3.4.  A 
compendium  of  the  these  quick  look  studies  is  not  included  in  this  document. 

Air-to-air  engagement  PkS  used  in  the  airairpk.dat  file  would  normally  be  derived 
using  Joint  Munitions  Effectiveness  Methodology  (JMEM)  data  and  results  of 
engagement  level  models  such  as  TAG  BRAWLER.  The  SIMAF  environment  will  allow 
the  user  to  make  TAG  BRAWLER  (or  other  engagement  level  and/or  mission  level 
model)  analysis  to  determine  missile  PkS  for  different  weapon/aircraft/tactics 
combinations.  These  PkS  will  be  used  to  calibrate  the  aggregate  weapons  platform  PkS 
used  in  THUNDER  simulations,  either  in  advance  of  running  the  simulation  or  on-the-fly 
as  “requested”  by  the  simulation.  The  notional  Pk  values  input  by  the  user  represent  those 
resulting  from  the  aggregation  of  inputs  coming  from  TAG  BRAWLER  or  a  similar 
model.  The  actual  Pk  values  used  in  the  simulation  were  chosen  primarily  to  show  a  delta 
with  the  addition  of  the  AIM-X  missile. 

Prior  to  the  initiation  of  experimental  runs  the  airairpk.dat  file  in  the  THUNDER 
scenario  provided  by  ASG/XR  were  modified.  The  original  file  yielded  a  Red  air-to-air 
2:1  kill  ratio  advantage  (~  400  Red  kills :  ~  200  Blue  kills)  for  a  30  day  war. 
Weapon/aircraft  PkS  in  the  airairpk.dat  file  were  changed  to  provide  approximately  equal 
air-to-air  kUl  ratios  (~200  kills/each)  for  Red  and  Blue  in  the  baseline,  F-15G,  case.  More 
detailed  discussion  of  the  experimental  design  is  included  in  section  3.6. 
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Ease  of  file  modification  was  facilitated  by  changing  parameters  in  existing 
aircraft  entities  rather  creating  new  entities  from  scratch.  The  existing  THUNDER  F-15C 
files  were  changed  to  F-XX  or  F-15C  with  updated  avionics  files.  AIM-7  files  were 
changed  to  AIM-X  files  and  AMRAAM  references  were  deleted  from  F-15C  and  F-XX 
data  files  as  required  for  the  alternative  solution  under  consideration.  Changing  from  one 
alternative  to  another  usually  involved  only  a  name  change,  e.g.  “F-15C”  to  “F-XX”,  and 
the  modification  of  one  two  parameters  in  the  existing  file  for  the  previous  aircraft  or 
missile  system.  Table  2  lists  the  THUNDER  data  files  modified  for  each  alternative. 

The  specific  changes  to  THUNDER  data  files  are  described  in  the  following  sections. 


Table  2,  Data  Files  Modified  by  Alternative  Solution 


F-15C 

w/Missile 

F-15C  w/Radar 

F-15C  w/Radar 
and  Missile 

F-XX 

F-XX 

w/Missile 

airairpk.dat 

airmunit.dat 

critres.dat 

relrngadvn.dat 

typeac.dat 

detect.dat 

typeac.dat 

typejam.dat 

typerdr.dat 

airairpk.dat 

airmunit.dat 

critres.dat 

detect.dat 

relrngadvn.dat 

typeac.dat 

typejam.dat 

typerdr.dat 

acserv.dat 

detect.dat 

squadron.dat 

typeac.dat 

typejam.dat 

typerdr.dat 

acserv.dat 

airairpk.dat 

airmunit.dat 

critres.dat 

detect.dat 

relrngadvn.dat 

squadron.dat 

typeac.dat 

typejam.dat 

typerdr.dat 

3.5.1F-XX  RCS  and  Radar  Modifications.  The  new  air  superiority  fighter,  the 
F-XX,  was  modeled  as  an  F-15C  with  a  very  low  RCS  and  an  improved  radar.  The 
reduced  RCS  impacts  detection  range  of  ground  and  early  warning  assets  in  THUNDER 
when  the  F-XX  is  the  only  type  aircraft  in  the  flight  group.  Otherwise,  THUNDER  uses 
the  RCS  of  the  aircraft  with  the  highest  RCS  in  the  flight  group.  One  of  the  air-to-air 
missions  captured  in  THUNDER  is  Barrier  Combat  Air  Patrol  (BARCAP).  BARCAP 
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aircraft  patrol  a  designated  area  on  their  side  of  the  Forward  Line  of  Troops  (PLOT)  to 
intercept  any  aircraft  that  attempt  to  pass  through  the  area  (THUINDER  Analyst  Manual, 
1995:21).  The  increased  radar  sweep  width  of  the  F-XX’s  radar  impacts  BARCAP 
detection  probabilities. 

Six  data  files  were  modified  to  implement  the  RCS  and  radar  improvements  that 
differentiated  the  F-XX  from  the  F-15C;  typeac.dat,  typerdr.dat,  typejam.dat,  detect.dat, 
acserv.dat,  and  squadron.dat.  Modifications  to  acserv.dat  and  squadron.dat  were  limited 
to  global  replacement  of  F-  15C  nomenclature  with  F-XX. 

Typeac.dat  required  the  addition  of  a  new  radar  cross  section  (RCS)  object  for  the 
F-XX  and  a  relabeling  of  the  F-15C  references.  An  RCS  of  0.05,  approximately  two 
orders  of  magnitude  below  the  baseline  F-15C  in  the  scenario,  was  chosen  for  F-XX 
RCS.  The  RCS  value  was  set  based  on  a  series  of  quick  look  analyses  of  the  effect  of 
RCS  of  Red  air-to-air  losses. 

The  F-XX’s  radar  was  implemented  by  adding  a  new  aircraft  radar  object  in  the 
typerdr.dat  file  for  the  aircraft  that  doubled  the  one  square  meter  detection  range  and 
maximum  range  of  the  radar  over  the  F-15C  system.  Additionally,  typerdr.dat  was 
modified  to  increase  the  sweep  width  of  the  original  F-15C  radar  in  the  file  from  90  to 
120  degrees  for  the  F-XX.  The  typejam.dat  file  required  similar  changes  as  those  to  the 
typerdr.dat  file,  the  creation  of  a  detect  object.  Although  the  scenario  did  not  require 
electronic  jamming,  the  typejam.dat  file  change  was  required  for  the  simulation  to  run. 
The  jamming  and  burn-through  profiles  for  the  F-XX  radar  were  left  at  the  same  as 
values  as  those  for  the  F-15C  radar. 
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Engagement  probability  (input  in  detect.dat)  as  used  in  THUNDER,  is  the 
probability  that  a  type  of  aircraft  will  be  able  to  engage  an  enemy  type  aircraft  given  a 
detection.  This  probability  is  an  aggregate  representation  of  the  factors  affecting  the 
outcome  of  an  aerial  engagement  once  initiated,  i.e.  sensor  suite,  cockpit  visibihty,  ability 
to  reposition  aircraft,  etc.  Detection  probabilities  are  a  function  of  airborne  early  warning 
(AEW)  state.  The  possible  states  represented  by  THUNDER  are  no  AEW,  Blue  AEW 
only.  Red  AEW  only,  and  both  Blue  and  Red  AEW  available  (Analyst  Manual,  1995:43). 

The  F-XX  RCS  and  radar  modifications  required  modifying  the  detect.dat  file  to 
reflect  both  a  decreased  RCS  and  an  increased  onboard  radar  capability.  The  probability 
of  RED  detecting  the  F-XX  in  each  of  the  four  AEW  scenarios  discussed  above  was  set 
to  probabilities  comparable  to  low  observable  (LO)  platforms,  the  Tomahawk  and  F-1 17, 
already  included  in  the  database.  The  probability  of  the  F-XX  radar  detecting  Red 
aircraft  was  increased  by  0.2  for  all  AEW  scenarios  to  characterize  the  improved 
detection  capability  of  the  new  radar. 

3.5.2  F-15C  Radar  Modifications.  Radar  modifications  to  the  F-15C  required 
the  changes  to  the  same  data  files  as  the  F-XX  radar  implementation;  typeac.dat, 
typerdr.dat,  typejam.dat,  and  detect.dat.  Typeac.dat  was  updated  for  the  new  radar 
identification  number.  The  same  changes  were  made  for  the  typerdr.dat  and  typejam.dat 
as  described  for  the  F-XX  in  the  previous  section.  The  detect.dat  only  required  the 
modification  of  the  probabilities  for  the  F-15C  detecting  Red  aircraft  in  each  of  the  four 
AEW  cases.  Hence,  detection  probabilities  were  increased  by  0.2  for  all  AEW  scenarios 
against  all  Red  aircraft  types. 
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3.5.3  F-XX/F-15C  Air-to-Air  Missile  Modifications.  As  noted  in  the 
introduction  to  this  section,  probability  of  kills  in  the  data  files  provided  by  ASC/XR 
were  initially  reduced  to  provide  a  relative  parity  (~  200  air-to-air  kills/side)  for  a  30  day 
war.  The  reduction  in  Pk’s  also  allowed  some  margin  to  implement  a  Pk  improvement  for 
the  introduction  of  a  new  missile.  The  new  missile,  the  AIM-X,  required  modification  of 
six  data  files  airairpk.dat,  airmunit.dat,  critres.dat,  relrngadvn.dat,  and  typeac.dat.  All 
AIM-7  instances  in  the  airmunt.dat  and  critres.dat  files  were  changed  to  AIM-X. 
Occiurences  of  the  AMRAAM  for  the  F-15C/F-XX  in  the  acserv.dat  and  typeac.dat  files 
were  changed  to  the  weapons  code  for  the  AIM-X. 

A  new  code  for  the  AIM-X  was  added  to  the  airairpk.dat  file.  The  new  missile 
was  included  in  the  configuration  for  the  F-15C/F-XX.  PkS  for  the  AIM-X  were  increased 
0.3  above  AMRAAM  PkS  against  all  Red  targets.  THUNDER  converts  the  raw  PkS  for 
each  weapon  into  an  aggregated  weapon  Pk  for  each  type  of  engaged  aircraft  (Analyst 
Manual,  1995:38).  The  aggregate  PkS  are  used  to  determine  the  attrition  rates  for 
defenders  and  attackers  in  aerial  engagements. 

The  advantage  gained  by  the  use  of  longer  range  weapons  is  represented  by  the 
concept  of  relative  range  advantage  (Analyst  Manual,  1995:  40).  This  advantage  impacts 
the  probability  that  an  attacking  aircraft  will  launch  a  weapon  successfully  at  a  defending 
aircraft.  A  new  killer  identification  number  was  added  to  the  relrngadvn.dat  file  for  the 
new  AIM-X  missile.  The  file  was  modified  to  reflect  a  AIM-X  range  advantage  over  all 
Red  ordnance. 
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3.6  Experimental  Design 

The  experiment  was  designed  as  a  two  way  analysis  of  variance  fixed  effects 
model.  The  first  treatment  was  the  F-15C  configuration.  As  previously  noted,  the  F-XX 
mirrors  the  F-15C  performance  with  a  reduced  RCS  and  an  improved  radar.  This 
treatment  was  fixed  at  three  levels  for  the  baseline  F-15C  and  the  two  alternative 
modifications.  The  other  treatment  was  the  armament  configuration.  Two  levels  of 
configurations  were  considered,  the  standard  F-15C  load  and  the  standard  load  with  the 
AIM-X  missiles  replacing  the  AMRAAMs.  The  design  yielded  six  treatment 
combinations.  The  experimental  design  for  the  simulation  is  depicted  in  Table  3. 


Table  3,  Experimental  Design 


Aircraft/ 

Ordnance 

F-15C 

F-15C  w/Radar 

F-15  w/RCS  & 
Radar  (F-XX) 

4  AMRAAM, 

4  AIM-9,  Gun 

4  AIM-X, 
4AIM-9,  Gun 

The  number  of  replications  at  each  design  point  was  determined  via  the  2-stage  method 
discussed  in  section  2.4.1. 

Common  random  numbers  (CRN)  or  correlated  sampling  is  often  applied  as  a 
variance  reduction  technique  (VRT)  for  the  comparison  of  two  or  more  alternate  system 
configurations  (Law  and  Kelton,  1991:613).  The  CRN  approach  permits  the  comparison 
of  alternate  systems  subjected  to  identical  or  nearly  identical  experimental  conditions  in 
hopes  of  differentiating  which  system  is  best  even  though  the  respective  estimates  are 
subject  to  sampling  error  (Goldsman,  1992:101).  Application  of  the  techniques  requires 
that  the  same  random  numbers  are  used  to  simulate  the  systems  of  interest  for  each 
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replication  of  an  experiment  (Banks,  Carson,  and  Nelson,  1996:481).  Employment  of 
CRN  in  the  AoA  study  implies  that  replication  12  of  the  THUNDER  simulation  for  the  F- 
XX  scenario  would  use  the  same  random  number  stream  as  replication  12  of  the  F-15C 
scenario,  the  F-15C  w/AIM-X  scenario,  etc.  Synchronization  of  the  random  numbers,  or 
forcing  the  same  random  numbers  to  be  used  for  the  same  purpose  at  each  of  the  design 
point,  can  enhance  the  effect  of  CRN  (Nelson,  1992:131). 

The  ten  distinct  random  number  generators  in  THUNDER  can  be  assigned  to 
different  event  categories,  e.g.  air  defense  events,  air-to-air  events,  etc.,  in  an  attempt  to 
maintain  positive  correlation  through  isolating  processes  within  the  simulation 
(THUNDER  Analyst  Manual,  1995:97).  However,  the  complexity  of  the  multitude  of 
interactions  in  the  model  contributes  to  the  uncertainty  of  the  variance  reduction  achieved 
through  the  synchronization  scheme.  The  THUNDER  simulations  in  this  effort  used  an 
unsynchronized  CRN  VRT  approach.  The  santK  random  number  seed  was  used  at  the 
initiation  of  the  runs  for  each  design  point,  with  no  further  efforts  taken  to  synchronize 
random  number  use. 

3.7  Simulation 

The  VV&A  of  THUNDER  for  use  in  this  experiment  began  with  the  modification 
of  data  files  for  alternative  scenarios  discussed  in  section  2.1.3.  The  simulation  was 
verified  through  debugging  the  changes  to  the  data  files  on  short,  one  to  two  replication, 
trial  simulation  runs.  Additional  verification  of  the  scenario  included  reviewing  the 
simulation’s  situation  map  (invoked  by  command  ttsm)  animation,  output  graphs 
(invoked  by  command  ttgraph),  and  output  reports  for  anomalies.  The  textual 
description  of  file  inconsistencies  and  errors  in  the  debug. out  file  generated  by 
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THUNDER  facilitated  the  verification  process.  The  verification  of  the  simulation 
included  an  active  dialogue  with  experienced  THUNDER  analysts  at  ASC/XR  and 
simulation  runs  under  a  variety  of  input  settings  for  reasonable  output  during  the  RCS 
exploration  discussed  in  section  2.2. 

A  conceptual  model  validation  of  the  scenario  for  this  experiment  was  performed 

by  comparing  THUNDER’S  assumptions,  limitations,  and  design  elements  against  the 

specific  requirements.  Validation  of  the  simulation’s  capability  to  provide  output  to  meet 

our  requirements  was  performed  in  parallel  with  the  verification.  The  experiment’s  focus 

on  modifying  scenario  data  to  provide  significant  differences  in  the  performance  of 

alternative  solutions  made  the  validation  and  accreditation  less  stringent  than  if  there  was 

a  requirement  for  high  fidelity  representations  of  each  player  in  the  simulation  to  evaluate 

actual  design  sensitivities.  The  author  accredited  the  model  for  the  thesis  effort  based  on 

literature  reviews  of  THUNDER  application  to  similar  efforts  and  personal  involvement 

in  the  validation  and  verification  discussed  in  the  paragraphs  above. 

The  air  war  was  simulated  in  THUNDER’S  high  resolution  mode.  This  setting 

provides  a  higher  granularity  representation  of  the  aircraft  combat  attrition  than  the 

binomial  distributed  flight  attritions  in  the  low  resolution  mode. 

In  the  high  resolution  mode,  flights  are  tracked  individually.  Stochastic 
determinations  are  made  as  to  whether  defender  and  flight  groups  enter  an 
engagement.  Outcomes  of  the  engagements  are  determined  based  on  escort 
tactics,  relative  range  advantages,  and  survival  and  kill  probability  data  for  the 
one-on-one  air  battles  between  the  various  engaging  threat  and  defender  aircraft 
types.  These  values,  the  number  of  aircraft  in  a  flight,  and  a  computed  probability 
of  such  an  engagement  occurring  between  flights  give  an  attrition  rate  for  the 
flight  (THUNDER  Analyst  Manual,  1995;  33). 
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3.8  Multivariate  Analysis 

The  initial  analysis  of  output  responses  for  the  F-15C,  F-15C  w/Missile,  F-15C 
w/Radar,  and  F-15C  w/Radar  &  Missile  alternatives  suggested  a  lack  of  statistical 
significance  between  the  alternate  solutions.  Consequently,  Bonferroni  confidence 
intervals  were  calculated  to  make  multiple  simultaneous  comparisons.  When  it  is  desired 
to  make  statements  about  several  variables  simultaneously,  the  Bonferroni  approach  is 
used  to  provide  the  analyst  with  a  fairly  high  confidence  all  statements  are  true 
simultaneously.  A  simultaneous  confidence  coefficient  of  at  least  (1  -  a)  can  be  assured 
by  choosing  the  confidence  intervals  I j  ,  for  j  =  1,2,  ...m  comparisons,  so  that 

2^-^^  =oc*  One  way  to  achieve  this  objective  is  if  each  interval  is  constructed  to  have 

confidence  coefficient  1-  (a  /m)  (Wackerly,  Mendenhall,  and  Schaeffer,  606:1996). 

The  smaller  the  value  of  Ofy ,  the  wider  the  j  th  confidence  interval.  The  major 

advantage  of  the  Bonferroni  approach  is  it  holds  whether  the  models  for  the  alternative 
designs  are  run  with  independent  sampling,  or  with  common  random  numbers  (Banks, 
Carson,  and  Nelson,  1996:493).  The  primary  disadvantage  is  that  as  the  number  of 
comparisons  increases,  the  widths  of  the  individual  intervals  increase.  The  Bonferroni 
technique  is  ordinarily  most  useful  when  the  number  of  simultaneous  estimates  is  not  too 
large  (Neter,  Kutner,  Nachtsheim,  and  Wasserman,  1996:155).  Ten  comparisons  is 
generally  recommended  as  the  hmit . 

The  confidence  intervals,  with  an  overall  confidence  coefficient  at  least 

(1- a)  are  given  by: 


(3.1) 
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where  Di  is  the  sample  mean  difference  averaged  over  all  replications  ( /? ),  is 

the  100*  (1  -  a,  /  2)  percentage  point  of  the  t  distribution  with  /?  - 1  degrees  of  freedom, 

s.e.ijbi )  is  standard  error  of  Di ,  and  m  is  the  total  number  of  comparisons  (Banks, 
Carson,  and  Nelson,  1996:494).  Confidence  intervals  completely  to  the  right  of  zero 
indicate  6^  >  0, .  Intervals  completely  to  the  left  of  zero  indicate  there  is  strong  evidence 
that  0)  <  0, .  Those  intervals  that  include  zero  offer  no  strong  evidence  that  one 
alternative  is  better  than  the  other. 

The  S AS. IMP  statistics  software  package  was  used  to  perform  statistical  analysis 
of  the  output  metrics  data.  The  distribution  of  each  case’s  data  was  assessed.  Since  the 
output  data  violated  normality,  a  statistical  procedure  was  required  that  did  not  require 
stringent  distributional  assumptions,  such  as  non-parametric,  or  distribution  free, 
methods.  Non-parametric  methods  are  inference  procedures  having  test  statistics  whose 
distribution  under  Ho  remains  the  same,  regardless  of  how  the  population  sampled  may 
change  (Larson  and  Marx,  510:1981).  The  Wilcoxon  signed  rank  test  is  a  non-parametric 
technique. 

In  performing  the  Wilcoxon  signed  rank  test,  the  signed  differences  between  the 
pairs  of  observations  are  rank  ordered  in  terms  of  their  absolute  size,  and  the  sign 
of  each  difference  is  attached  to  the  rank  associated  with  that  difference  is 
attached  to  the  rank  associated  with  that  difference.  The  test  statistic  is  then  W, 
the  sum  of  the  ranks  with  the  less-frequently-occurring  sign.  For  large  n ,  the 
sampling  distribution  of  W  is  approximately  normal  (Winkler  and  Hays, 
856:1975). 

The  signed  rank  test  requires  equal  samples  sizes.  No  assumptions  are  made  concerning 
the  underlying  population  distributions.  A  large-sample  (n  >25)  Wilcoxon  signed  rank 
test  was  used  on  the  matched  paired  observations  of  the  F-15C  and  the  F-XX  w/missile 
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(the  extremes  of  the  design  points)  in  the  investigation  to  test  the  hypothesis  that  the 
outputs  for  the  two  alternatives  have  the  same  distributions  (Wackerly,  Mendenhall,  and 
Schaeffer,  1996:656).  Failure  to  reject  the  hypothesis  for  the  two-tailed  test  was  used  as 
a  basis  to  discard  an  output  as  statistically  insignificant.  The  variance-covariance  and 
correlation  matrices  of  the  six  scenarios  were  calculated  to  determine  the  variances  and 
correlations  in  the  remaining  data  for  each  case. 

The  data  was  then  standardized.  For  the  univariate  case,  standardized  scores  in 
excess  of  +  3.0  were  considered  outliers  (Tabachnick  &  Fidell,  1989:  68).  Those  runs 
with  standardized  scores  in  excess  of  ±  1.9  for  more  than  one  response  were  considered 
multivariate  outliers.  The  outlier  analysis  was  not  to  eliminate  data  points.  The  outlier 
information  was  used  primarily  for  insights  on  the  data  structure. 

A  Principal  Component  Analysis  (PCA)  was  performed  on  the  data  to  determine 
the  dimensionality.  Component  loadings  were  determined.  The  different  units 
represented  by  the  outputs  necessitated  the  use  of  the  correlation  matrix  and  standardized 
data  for  PCA  extraction.  Two  graphical  dimensionality  tests  were  performed  -  magnitude 
of  the  eigenvalues  (Kaiser’s  test)  and  the  scree  test.  The  dimensionality  insights  from  the 
PCA  were  used  to  determine  a  starting  point  for  the  rotations  of  factors  in  the  factor 
analysis. 

A  cluster  analysis  was  performed  on  the  standardized  data  as  a  discovery  tool  for 
the  potential  groupings  of  the  data.  Agglomerative  clustering  methods  were  applied  to 
the  standardized  data.  Data  was  grouped  via  the  squared  Euclidean  distance  between  the 
means  of  the  clusters.  The  grouping  of  the  clusters  provided  additional  insight  into  the 
dimensionality  and  structm-e  of  the  data. 
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An  exploratory  Factor  Analysis  (FA),  using  the  dimensionality  information 
gleaned  from  the  PCA  and  cluster  analyses,  was  performed  to  determine  a  common 
structure  underlying  the  data.  As  with  the  PCA,  different  units  represented  by  the 
THUNDER  measured  outputs  necessitated  use  of  the  correlation  matrix  for  FA 
extraction.  The  dimensionality  data  furnished  insights  on  the  starting  point  for 
performing  iterative  rotations  of  the  components  via  the  varimax  rotation  n^thod. 

Successive  rotations  of  the  factors  yielded  an  interpretable  loading  matrix.  A 
residual  matrix  of  the  final  loadings  was  calculated  to  assess  the  fit  of  the  factor  model. 
The  factors  were  then  interpreted  based  on  the  final  loadings.  The  final  factor  loading 
matrix  was  used  to  estimate  the  factor  coefficients  score  matrix  for  additional  insights  on 
the  data  structure  via  visualization  of  the  scores. 

3.9  Visualization 

The  visualization  effort  had  two  purposes,  1)  employ  a  visual  paradigm  for 
reducing  MOE  dimensionality,  and  2)  develop  an  AoA  visualization  approach  for  the 
SIMAF  capability  demonstration.  Notional  LCCs  were  used  for  all  alternatives  in 
developing  the  graphical  representation  of  cost/effectiveness  integration  step  of  the  AoA 
study  effort  discussed  in  section  2.1.4. 

Visualization  of  the  data  was  via  MineSet  3.0,  a  data  mining  and  visualization 
tool  produced  by  Silicon  Graphics  Incorporated  (SGI).  The  visualization  tool  used 
required  development  of  a  data  file  consisting  of  rows  of  tab-separated  fields  (“save  as” 
option  in  EXCEL)  and  a  configuration  file  (data  schema)  describing  the  format  of  the 
input  data  (MineSet  User’s  Guide,  1997).  The  principal  MineSet  tool  used  in  the  analysis 
for  the  discovery  of  data  patterns  and  trends  was  the  “Scatter  Visualizer”  depicted  in 
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Figure  7.  The  tool  was  used  to  display  the  data  via  a  three  dimensional  scatter  plot 
paradigm .  Additional  dimensionality  was  added  to  the  rendering  of  the  data  via  the  use 


Figure  7,  MineSet  Scatter  Visualizer  (MineSet  User’s  Manual,  1997) 
of  color,  size,  and  entity  labeling.  Filtering,  rotation  of  the  data,  and  focus  on  individual 
data  points  were  used  to  drUl  down  into  the  data  and  visually  data-mine  for  structural 
insights.  The  KDD  process  outlined  in  Figure  5  was  used  as  a  guideline  in  visualization 
development  and  data  mining  application  for  the  thesis  effort. 

Since  the  goal  of  the  end-user  was  to  gain  insights  on  the  AoA  from  a  multivariate 
perspective,  the  visual  data  mining  effort  targeted  two  areas:  the  output  responses  of  the 
simulation  and  the  factor  scores  from  the  multivariate  analysis  of  the  output  responses 
discussed  in  this  chapter.  The  simulation  output  data  was  complete  and  clean  (no 
spurious  symbols,  illegal  operations,  etc.).  Consequently,  data  cleaning  and 
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preprocessing  was  not  required.  As  noted  previously  in  the  chapter,  tests  of  statistical 
significance  were  applied  to  reduce  the  number  of  dependent  variables  under 
consideration.  Additionally,  estimates  of  the  centroids  for  each  of  the  for  the  alternative 
output  responses  and  factor  scores  was  included  in  the  data  set  to  facilitate  filtering.  Data 
mining  algorithms  were  not  applied  to  the  data  using  the  tools  organic  to  the  MineSet 
application.  Rather  FA  data  reduction  algorithms  were  performed  off-line  and  the 
resulting  factor  scores  imported  to  MineSet  for  viewing.  The  data  mining  effort  focused 
on  patterns  of  data  for  the  respective  alternatives  clustering  at  specific  output  response 
and  factor  score  levels  and  the  dispersion  (variability)  of  those  clusters.  Finally,  iterative 
interactions  with  the  data  utilizing  the  steps  described  above  yielded  discovered 
knowledge  on  output  response  structure,  factor  score/data  dimensionality,  and 
AoA  cost/effectiveness  integration  that  is  documented  in  the  results  and  conclusions 
sections  of  this  report,  chapters  four  and  five. 

Once  the  analyst  completed  the  visual  data-mining  and  analysis.  The  decision 
maker,  autonomously  and  assisted  by  the  analyst,  interacted  with  the  data  using  the  same 
or  a  subset  of  the  visualization  tools  employed  by  the  analyst.  The  MineSet  software 
permitted  the  decision  maker  to  customize  the  depth  of  data  mining  and  KDD  to  their 
personal  comfort/interest  level,  supporting  either  a  high  level  panning  and  zooming 
through  the  data  focusing  on  specific  points  of  interest,  or  a  detailed  “what  if’ 
investigation  of  the  data  set  utilizing  real-time  filtering  or  other  data  mining  tools.  Based 
on  the  presentation  of  the  analytical  results  and  the  KDD  gained  by  the  visualization,  the 
decision  maker  was  equipped  to  make  an  acquisition  decision  or  to  scope  additional  data 
gathering  efforts  to  facilitate  a  decision. 
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4.  Results 


This  chapter  discusses  the  analysis  of  the  experimental  design,  the  multivariate 
analysis  and  the  visualization  results.  Symmetric  matrices  in  this  section  are  displayed  as 
a  diagonal  and  lower  matrix  elements. 

4.1  Experimental  Design  Analysis 

Ten  simulation  output  responses  were  measured.  The  output  responses  and  the 
abbreviations  for  the  responses  in  this  document  are  listed  in  Table  4. 


Table  4,  Description  of  Output  Response  Measures 


AirSup 

Days  to  air  supremacy  (10%  of  initial  Red  sortie  rate) 

BLossSort 

Blue  air-to-air  losses  per  blue  sortie  flown 

BlueGrnd 

Total  number  of  Blue  aircraft  destroyed  on  the  ground 

BlueSA 

Blue  aircraft  losses  to  enemy  SAMs  per  Blue  sortie  flown 

RedACQ 

Number  of  Red  acquisition  radars  destroyed 

RedFC 

Number  of  Red  fire  control  radars  destroyed 

RedLoss 

Red  aircraft  lost  to  Blue  air 

RedOpen 

Number  of  Red  aircraft  destroyed  at  their  home  airfields  in  the  open 
(not  sheltered) 

RedSortSO 

Red  sortie  generation  capability  on  day  30  (ratio  to  initial  sortie 
generation  rate) 

RedTEL 

Number  of  Red  transportable  erector/launchers  destroyed 

Initially,  four  independent  replications  were  made  at  each  design  point.  These  initial 
sample  sizes  of  four  were  used  to  determine  the  final  sample  size  (and  the  remaining 
number  of  replications)  required  to  give  a  user  specified  precision  with  a  90%  confidence 
level.  Based  on  anticipated  absolute  error  thresholds  to  discriminate  between  the 
alternatives  for  each  output  response  and  simulation  time  required  for  each  replication  (~ 
0.5  hours),  thirty  one  replications  were  performed  at  each  design  point.  The  estimated 
90%  confidence  interval  half-lengths  for  each  response  and  design  point  based  on  thirty 
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one  replications  of  the  experiment  and  the  standard  deviations  of  the  initial  samples  are 
listed  in  Appendix  E.  The  output  responses  are  at  Appendix  F. 

Relative  frequency  histograms  of  the  response  data  for  each  of  the  design  points 
was  plotted  for  a  graphical  description  of  the  data.  The  histograms  for  the  responses  are 
in  Appendix  G.  The  rectangles  on  the  histograms  indicate  the  fraction  of  total  output 
measurements  falling  within  each  range  indicated  on  the  x-axis. 

The  mean  output  responses  and  confidence  intervals  for  each  design  point  are 
listed  in  Table  5.  Discussion  of  statistical  significance  in  this  chapter  refers  to 
Table  5,  Means  of  Output  Responses  w/90%  Confidence  Intervals 


MOEs 

Aircraft/ 

Weapons 

F-15C 

F-15C  w/Radar 

F-15Cw/RCS& 
Radar  (F-XX) 

RedLoss 

Standard  Load 

214  +  6 

317  +  5 

RedSortSO 

0.0404±0.0036 

0.0316+0.0039 

0.0093+0.0012 

BLossSort 

0.0028±0.0001 

Mmmmm 

0.0017+0.0001 

AirSup 

BlueSA 

EMMSm 

BlueGmd 

99  +  4 

91+5 

81  +  4 

242  +  5 

242  +  6 

181  +  5 

RedTEL 

RedACO 

44  +  21 

39+13 

^BSSBSBB 

RedFC 

RedLoss 

New  Missile 

227  +  8 

253  +  7 

359  +  6  1 

RedSortSO 

BLossSort 

mmmmm 

AirSup 

BlueSA 

mmmmm 

mmmmm 

BlueGmd 

90  +  4 

87  +  5 

73  +  3  1 

ISSS7S3HI 

231+7 

216  +  4 

1  RedTEL 

33  +  10 

1  RedFC 

overlapping  output  response  90%  confidence  interval  half  lengths  between  alternatives  as 


indicating  a  lack  of  statistical  significance.  The  mean  of  the  RedLoss  response  increases 
with  both  modifications  to  the  current  air  superiority  fighter  (radar  and  reduced  RCS)  and 
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the  new  missile  indicating  these  changes  increase  the  air-to-air  lethality  of  the  fighter. 

The  radar  improvements  appear  to  cause  a  small  increase  in  RedLoss  (~10  -  25),  the  new 
missile  causes  a  moderate  change  (~20  -  40),  and  the  RCS  reduction  causes  a  large 
change(~100).  The  F-15C  and  F-15C  w/Radar,  and  F-15C  w/Radar  and  F-15C  w/Missile 
are  not  statistically  different  for  RedLoss  responses.  RedSortSO,  BLossSort,  AirSup, 
BlueSA,  BlueGmd,  and  RedOpen  responses  decreased  as  the  modifications  to  the  F-15C 
were  implemented.  For  all  six  of  these  measures,  the  most  pronounced  changes  occm  at 
the  reduced  RCS  design  points.  RedSortSO  and  AirSup  have  the  same  interpretation  as 
the  RedLoss.  As  the  air-to-air  lethality  of  the  alternative  increases  the  rollback  of  Red 
sortie  generation  occurs  more  rapidly.  F-15C  w/Missile  and  F-15C  w/Radar  are  not 
statistically  different  for  either  RedSortSO  or  AirSup.  F-15C  w/Radar  and  F-15C  w/Radar 
and  Missile  are  not  statistically  different  for  RedSortSO.  BLossSort  and  BlueGrnd 
responses  indicate  with  less  Red  fighters  airborne,  there  are  less  shooters  to  attrit  Blue 
either  in  the  air  or  on  the  ground.  BLossSort  and  BlueGmd  responses  for  F-15C 
w/Radar,  F-15C  w/Missile,  and  F-15C  w/Radar  and  Missile  are  not  statistically  different. 
Decreasing  RedOpen  responses  corresponding  to  F-15C  improvements  indicate  that  as 
more  Red  aircraft  are  attrited,  the  Offensive  Counterair  missions  targeting  fighters  in  the 
open  have  less  targets  to  choose  from.  There  is  no  difference  between  the  F-15C  and  the 
F-15C  w/Radar  for  RedOpen  responses.  There  was  not  as  great  a  difference  in  the 
BlueSA  output  between  design  points  as  for  the  other  responses.  BlueSA  responses  were 
not  statistically  different  for  a  comparison  between  the  F-15C,  the  F-15C  w/Missile,  the 
F-15C  w/Radar,  or  between  the  F-15C  w/Radar  and  Missile  design  points,  or  between  the 
F-15C  w/Radar  and  the  F-XX.  The  increase  of  RedFC  responses,  albeit  small,  indicated 
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the  fighter  modifications  improved  the  effectiveness  of  suppressing  enemy  air  defenses. 
There  was  no  statistical  difference  between  the  F-15C,  the  F-15C  w/Radar,  and  the  F-15C 
w/Radar  and  Missile,  or  between  the  F-15C  w/Missile,  the  F-15C  w/Radar,  the  F-15C 
w/Radar  and  Missile,  and  the  F-XX,  or  between  the  F-XX,  the  F-XX  w/Missile,  and  the 
F-15C  w/Missile. 

The  number  of  cases  for  which  the  output  responses  were  the  same  between  the 
different  F-15C  alternatives  (radar,  new  missile,  both)  indicates  the  performances  of  the 
platforms  at  the  campaign  level  of  analysis  are  very  similar.  The  Bonferroni  confidence 
intervals  described  in  chapter  three  were  calculated  to  simultaneously  compare  the  F- 
15C,  F-15C  w/Missile,  F-15C  w/Radar,  andF-15C  w/Radar  &  Missile  alternatives  for 
each  output  response.  Appendix  H  depicts  the  Bonferroni  90%  confidence  intervals  for 
F-15C,  F-15C  w/Missile,  F-15C  w/Radar  and  F-15C  w/Radar  &  Missile  output  response 
comparisons.  RedLoss,  RedSortSO,  BLossSort,  AirSup,  BlueGmd,  RedOpen,  RedFC, 
and  RedTEL  output  responses  indicate  at  least  one  comparison  with  a  statistically 
significant  finding. 

The  RedLoss  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Missile  and  F-15C 
w/Radar  &  Missile  are  left  of  zero  indicating  the  F-15C  performance  in  attriting  red 
aircraft  improves  with  the  addition  of  the  new  missile  and  the  improved  radar/new 
missile  combination.  F-15C  w/Missile  vs.  F-15C  w/Radar  &  Missile  is  also  left  of  zero 
suggesting  higher  red  aircraft  attrition  for  the  Blue  forces  with  the  improved  radar/new 
missile  pairing  versus  the  new  missile  alone.  F-15  w/Radar  output  response  performance 
is  less  than  F-15C  w/Radar  &  Missile.  F-15C  vs  F-15C  w/Radar  and  F-15C  w/Missile 
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vs.  F-15C  w/Radar  do  not  offer  strong  evidence  that  one  alternative  is  better  than  the 
other. 

The  RedSortSO  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Missile,  F-15C 
w/Radar,  and  F-15C  w/Radar  &  Missile  are  all  right  of  zero  indicating  the  improvements 
to  the  current  air  superiority  fighter  improve  Blue  force’s  capability  to  draw  down  Red 
sortie  generation.  F-15C  w/Missile  vs  F-15C  w/Radar  and  F-15C  w/Radar  &  Missile, 
and  F-15C  w/Radar  vs.  F-15C  w/Radar  &  Missile  do  not  offer  strong  evidence  that  one 
alternative  is  better  than  the  other. 

The  BLx)ssSort  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Missile  and  F-15 
w/Radar  &  Missile  are  right  of  zero  indicating  the  new  missile  and  the  improved 
radar/new  missile  combination  improve  the  F-15C’s  capability  to  reduce  Blue  air-to-air 
losses  per  sortie.  F-15C  vs  F-15C  w/Radar,  F-15C  w/Missile  vs.  F-15C  w/Radar,  and  F- 
15C  w/Radar  vs.  F-15C  w/Radar  &  Missile  do  not  offer  strong  evidence  that  one 
alternative  is  better  than  the  other. 

The  AirSup  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Missile,  F-15C 
w/Radar,  and  F-15C  w/Radar  &  Missile  are  right  of  zero  indicating  the  improvements  to 
the  F-15C  reduces  the  number  of  the  days  for  Blue  forces  to  reach  air  supremacy. 
Additionally,  F-15C  w/Missile  vs.  F-15C  w/Radar  &  Missile  suggest  the  improved 
radar/new  missile  combination  has  a  greater  effect  on  Blue  forces  acheiving  air 
supremacy  than  the  new  missile  alone.  F-15C  w/Radar  vs.  F-15C  w/Radar  &  Missile  is 
also  right  of  zero  suggesting  the  pairing  of  the  improved  radar  with  the  new  missile  is  an 
improvement  over  the  radar  alone  for  reducing  the  days  to  air  supremacy.  Only  the 
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F-15C  w/Missile  vs.  the  F-15C  w/Radar  does  not  seem  to  provide  statistical  evidence  that 
one  alternative  is  better  than  the  other. 

Only  the  BlueGrnd  confidence  interval  for  the  F-15C  vs.  F-15C  w/Radar  & 

Missile  is  right  of  zero  indicating  the  radar  improvement  and  new  missile  combination 
has  a  greater  effect  reducing  Blue  losses  on  the  ground  than  the  current  F-15C.  The 
remainder  of  the  BlueGrnd  confidence  intervals  suggest  no  strong  statistical  evidence  that 
one  alternative  is  better  than  the  other. 

The  RedOpen  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Radar  &  Missile  is 
right  of  zero  indicating  the  radar  improvement  and  the  new  missile  pairing  reduces  the 
Red  aircraft  destroyed  in  the  open.  Also,  F-15C  w/Missile  vs.  F-15C  w/Radar  &  Missile 
is  right  of  zero  indicating  the  radar  improvement  and  new  missile  configuration  has  a 
greater  effect  on  reducing  Red  aircraft  destroyed  in  the  open  than  the  new  missile  alone. 
Additionally,  the  F-15C  w/Radar  vs.  F-15C  w/Radar  &  Missile  indicates  the  radar/missile 
combination  results  in  less  Red  aircraft  destroyed  in  the  open  than  the  radar  alone.  This 
tracks  with  the  original  interpretation  of  the  information  in  Table  4  -  increased  RedLoss 
performance  leaves  less  Red  targets  in  the  open.  F-15C  vs.  the  F-15C  w/Radar  and 
Missile,  F-15C  w/Missile  vs.  F-15C  w/Radar  &  Missile,  and  F-15C  w/Radar  vs.  F-15G 
w/Radar  &  Missile  do  not  seem  to  provide  statistical  evidence  that  one  alternative  is 
better  than  the  other. 

The  RedFC  confidence  intervals  for  the  F-15C  vs.  F-15C  w/Missile  is  the  only 
interval  to  the  left  of  zero  suggesting  the  new  missile  increases  the  number  of  Red  fire 
control  radars  destroyed.  The  remainder  of  the  RedFC  confidence  intervals  suggest  no 
strong  statistical  evidence  that  one  alternative  is  better  than  the  other. 
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The  RedTEL  confidence  interval  for  the  F-15C  vs.  F-15C  w/Missile  is  the  only 
interval  to  the  left  of  zero  indicating  the  new  missile  makes  Blue  forces  more  effective  at 
destroying  Red  TELs  than  the  current  air  superiority  aircraft.  The  remainder  of  the 
RedTEL  confidence  intervals  suggest  no  strong  statistical  evidence  that  one  alternative  is 
better  than  the  other. 

Both  the  BlueSA  and  the  RedACQ  families  of  confidence  intervals  include  zero 
for  all  alternatives  suggesting  there  is  no  strong  statistical  evidence  that  one  system 
design  is  better  than  another  for  those  output  response  categories. 

Table  6  summarizes  the  information  from  the  Bonferroni  confidence  intervals. 
The  cells  are  shaded  for  the  comparisons  where  the  confidence  intervals  suggest  strong 
statistical  evidence  that  the  first  alternative  listed  is  better  than  the  second.  The  empty 
cells  depict  those  cases  where  the  confidence  intervals  indicate  strong  statistical  evidence 
the  first  alternative  listed  is  inferior  to  the  second.  In  the  case  where  the  confidence 
intervals  included  zero,  a  “?”  is  listed  in  the  table  cell.  Trends  in  the  table  indicate  the  F- 
15C  w/Missile  alternative  is  better  than  the  F-15C  alternative  (six  of  ten  output 
responses),  the  F-15C  w/Radar  alternative  is  better  than  the  F-15C  alternative  (two  of  ten 
output  responses),  the  F-15C  w/Radar  &  Missile  alternative  is  better  than  the  F-15C 
alternative(six  of  ten  responses),  the  F-15C  w/Radar  &  Missile  alternative  is  better  than 
the  F-15C  w/Missile  alternative  (three  of  ten  responses),  and  the  F-15C  w/Radar  & 
Missile  alternative  is  better  than  the  F-15C  w/Radar  (two  of  ten  responses). 
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Table  6,  Summary  of  Bonferroni  Family  of  Confidence  Intervals 
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A  Wilcoxon  Signed-Rank  Test  was  performed  between  the  F-15C  and  the  F-XX 
with  the  new  missile  design  points.  The  test  was  performed  to  evaluate  the  hypothesis 
that  the  medians  of  the  output  responses  were  equal.  Failure  to  reject  the  null  hypothesis 
indicates  the  output  responses  for  a  particular  measure  may  not  be  significant.  The 
results  of  the  signed-rank  test  are  in  Table  7. 

The  high  signed-rank  p- values  for  the  RedTEL  and  RedACQ  output  responses 
indicate  the  distributions  of  the  two  fields  are  the  same  for  the  F-15C  and  F-XX 
w/Missile  alternatives.  This  suggests  that  at  a  90%  confidence  interval,  the 
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Table  7,  Wilcoxon  Signed-Rank  Test  for  F-15C  and  F-XX  w/New  Missile 


CO 

o 

1 

CO 

■S 

CO 

CO 

CO 

O 

AirSup 

< 

1 

1 

1 

1 

O' 

o 

PQ 

PQ 

pq 

pti 

p^ 

Pi 

Two-tailed  p  value 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.094 

0.860 

0.000 

Wilcoxon  Z  value 

-4.860 

-4.860 

-4.860 

-4.860 

-4.223 

-4.782 

-4.860 

-1.685 

-0.176 

-3.645 

employment  of  the  F-XX  w/Missile  does  not  increase  the  number  of  Red  acquisition 
radars  or  TELs  destroyed  by  Blue  forces.  The  Air-to-Air  Escort  (AIRESC)  and  Fighter 
Sweep  (FSWP)  air  superiority  taskings  of  the  alternatives  in  the  study  have  a  negligible 
effect  on  the  outcome  of  the  air-to-ground  interdiction  sorties  that  target  Red  acquisition 
radars  and  TELs.  AIRESC  missions  accompany  missions  to  provide  air-to-air  protection. 
FSWP  missions  attack  enemy  aircraft  that  are  operating  on  their  side  of  the  PLOT  during 
a  period  when  friendly  ground  attack  aircraft  are  operating  in  the  vicinity  (THUNDER 
Analyst  Manual,  20:1995).  Since  F-15C  and  F-XX  w/Missile  were  used  in  the  signed- 
rank  test  to  bound  the  alternative  output  responses,  ehminating  RedTEL  and  RedACQ 
output  responses  from  the  multivariate  analysis  should  not  effect  the  results.  Table  8  lists 
a  variance-covariance  matrix,  C ,  of  the  remaining  eight  output  responses. 


Table  8,  Variance-Covariance  Matrix  of  Output  Responses 


Variables 

RedLoss 

RedSort30 

BLossSort 

AirSuD 

BlueSA 

BlueGrnd 

RedOpen 

RedFC 

RedLoss 

3625 

RedSortao 

-0.838 

2.83E-04 

BLossSort 

-0.03 

7.99E-06 

3.305E-07 

AirSuD 

-151.783 

0.044 

0.001 

8.66 

BlueSA 

4.57E-06 

1.533E-07 

7.98E-04 

1.953E-07 

BlueGrnd 

-427.029 

0.137 

0.005 

24.115 

0.003 

248.326 

RedOoen 

-1961 

0.382 

0.015 

73.074 

0.006 

210.499 

1241 

RedFC 

7093 

-2.599 

-0.092 

-438.849 

-0.105 

-2128 

-3028 

209400 
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The  variances  are  along  the  diagonal  of  the  matrix.  The  wide  range  of  variances 
and  diverse  units  of  the  output  data  indicates  the  correlation  matrix,  R ,  is  the  best  choice 


for  extracting  the  PCA  and  FA.  R  for  the  eight  output  responses  is  depicted  in  Table  9. 


Table  9,  Correlation  Matrix  of  Output  Responses 


IVariable 

RedLoss 

BLossSort 

AirSuD 

BlueSA 

BlueGrnd 

RedOoen 

RedFC 

RedLoss 

1 

-0.8269 

1 

BLossSort 

-0.8679 

0.8264 

1 

AirSuo 

-0.8567 

0.8837 

0.878 

1 

BlueSA 

-0.551 

0.6149 

0.6035 

0.6139 

1 

BlueGrnd 

-0.4501 

0.5182 

0.5349 

0.52 

0.413 

1 

RedOpen 

WEMm 

0.6441 

0.7514 

0.705 

0.3981 

0.3792 

1 

RedFC 

0.2574 

-0.3375 

-0.3504 

-0.3258 

-0.5192 

-0.2951 

1 

The  correlation  matrix  indicates  a  significant  negative  correlation  (>  .5)  between 

RedLoss  and  RedSortSO,  BLossSort,  AirSup,  BlueSA,  and  RedOpen.  All  these 

correlations  seem  intuitive  except  the  negative  loading  on  BlueSA. 

The  description  of  high  resolution  air  defense  site  engagements  in  the  THUNDER 

analyst  manual  offers  an  explanation  of  the  BlueSA  correlations. 

In  HIGH  resolution,  the  distance  at  which  an  air  defense  site  wiU  detect  the  group 
is  based  upon  the  site’s  acquisition  and  fire  control  capabilities,  the  flight  group’s 
radar  cross  section,  ingress  and  terminal  altitudes,  jamming  ability,  and  the 
terrain.  Once  the  site  is  encountered,  THUNDER  uses  the  acquisition  distance 
and  weapon  range  to  determine  the  number  of  rounds  the  site  can  fire.  These 
factors,  along  with  the  Pk,  determine  the  flight  group’s  losses  (THUNDER 
Analyst  Manual,  55:1995). 

Air  defense  site  Pk’s  were  not  modified  for  the  different  alternatives  in  this  investigation. 
In  the  HIGH  resolution  mode  of  THUNDER,  the  reduced  RCS  of  the  F-XX  should 
increase  red  aircraft  losses  and  reduce  blue  surface-to-air  losses/sortie.  The  BlueSA 
output  responses  for  the  F-15C,  F-15C  w/Missile,  F-15C  w/Radar,  and  F-15C  w/Radar  & 
Missile  should  not  change  significantly.  This  conjecture  is  supported  by  the  family  of 
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Bonferroni  confidence  intervals  for  BlueS  A  in  Table  6  that  all  include  zero  and  the 
results  of  the  Wilcoxon  signed-rank  test  for  F-15C  and  the  F-XX  w/Missile  BlueS  A 
output  responses  in  Table  7. 

RedSortBO  has  significant  correlations  with  RedLoss,  BLossSort,  AirSup, 

BlueSA,  BlueGmd,  and  RedOpen.  All  correlations  are  fairly  intuitive  except  for  the 
BlueS  A  just  discussed.  BLossSort  has  significant  correlations  with  RedLoss,  RedSortSO, 
AirSup,  BlueSA,  BlueGrnd,  and  RedOpen.  AirSup  has  significant  correlations  with 
RedLoss,  RedSortSO,  BLossSort,  BlueSA,  BlueGmd,  and  RedOpen.  BlueSA  has 
significant  correlations  with  RedLoss,  RedSortBO,  BLossSort,  AirSup,  and  RedFC. 

Again  the  correlations  are  fairly  intuitive.  BlueGmd  has  significant  correlations  with 
RedSortSO,  BLossSort  and  AirSup.  RedOpen  has  significant  correlations  with  RedLoss, 
RedSortSO,  BLossSort,  and  AirSup.  RedFC  has  significant  correlations  with  BlueSA. 

Multivariate  analysis  was  performed  on  the  eight  MOE  responses;  RedLoss, 
RedSortSO,  BLossSort,  AirSup,  BlueSA,  BlueGrnd,  RedOpen,  and  RedFC. 

4.2  Multivariate  Analysis 

4.2.1  Outliers.  Prior  to  any  applications  of  multivariate  data  reduction  tools,  the 
data  was  standardized  and  an  analysis  of  outliers  ( >  1.9)  was  performed.  The  significant 
outliers  by  case  and  replication  are  listed  in  Table  10.  The  alternatives  averaged  about 
three  replications  with  potential  outliers.  The  F-15C  w/Radar  had  the  least  replications 
with  potential  outliers  (two)  and  the  F-XX  had  the  most  (six).  Outliers  were  generally 
limited  to  one  or  two  variables  for  each  affected  replication.  No  outliers  were  excessive  - 
there  were  no  replications  with  multiple  responses  »2  (Tabachnick  &  Fidell,  1989:68). 
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Table  10,  Standardized  Outliers  by  Alternative  and  Replication 


Scenario 

■ 

Red  A-A 
Losses 

Red 

Percent 

Sortie 

Blue 

Loss 

/Sortie 

Air 

Supremacy 
_ (-10) 

Blue  S-A 
Losses 
/Sortie 

Total  Blue 
Destroyed 
on  Ground 

Total  Red 
Destroyed 
in  the 
Open 

Total 

RedFC 

Radars 

Killed 

F-15C 

4 

3.0 

msm 

■■■■ 

F-15C 

19 

-2.2 

m^m\ 

F-15C 

23 

2.1 

-2.3 

F-15C  Msl 

3 

2.1 

-2.2 

Mill  1  wtm 

10 

3.0 

1.9 

F-15C  Msl 

25 

2.2 

-2.0 

F-15C  Rdr 

26 

2.0 

2.0 

-2.1 

F-15C  Rdr 

27 

-2.3 

-2.1 

F-15CRdr/Msl 

12 

2.0 

F-15C  Rdr/MsI 

24 

-2.2 

2.0 

F-XX 

3 

2.8 

-2.1 

F-XX 

12 

3.1 

F-XX 

9 

2.4 

■il^H 

■■m 

2.1 

F-XX 

19 

-2.0 

2.2 

F-XX 

21 

1.9 

2.1 

-2.1 

F-XX 

25 

-3.1 

F-XX  Msl 

1 

■^i 

2.0 

F-XX  Msl 

14 

4.2 

mu 

F-XX  Msl 

18 

1.9 

1 

I 

1 

-2.1 

F-XX  Msl 

19 

-2.4 

■■■ 

-2.2 

1 _ 

Based  on  the  outlier  analysis,  the  output  responses  of  all  replications  were  retained  for  the 


remaining  analysis. 

4.2.2  PC  A.  PCA  loadings,  depicted  in  Table  1 1,  were  extracted  from  the 
correlation  matrix.  The  eigenvalues  of  each  component  are  in  the  top  row  of  the  matrix. 


Table  11,  PCA  Loading  and  Variance  Reduction  of  Output  Responses 


EiqenValue: 

5.1929 

1.0811 

0.6631 

0.4566 

0.323 

0.1547 

0.0998 

0.0288 

Percent: 

64.9 

13.5 

8.3 

mmm 

4.0 

1.9 

1.2 

0.4 

CumPercent: 

64.9 

78.4 

86.7 

92.4 

96.5 

98.4 

99.6 

100.0 

^bhhi 

RedLoss 

-0.4083 

-0.2698 

0.1546 

0.1494 

0.1636 

0.0331 

0.8168 

RedSortao 

0.3980 

0.0450 

0.0068 

0.2728 

-0.4764 

0.2110 

BLossSort 

0.4092 

0.0912 

-0.0174 

0.0126 

-0.1582 

0.7832 

-0.4250 

0.0690 

AirSup 

0.4084 

0.0905 

-0.0140 

0.1934 

-0.3584 

0.1258 

0.7977 

0.0828 

BlueSA 

0.3159 

-0.1804 

0.5829 

0.5890 

0.0033 

0.0488 

BlueGmd 

0.2732 

0.1587 

0.0201 

-0.0329 

RedOpen 

0.3567 

0.3668 

0.4327 

0.0510 

RedFC 

0.7479 

0.2784 

0.5296 

0.2020 

0.0384 

0.0266! 

The  individual  variance  reduction  provided  by  each  component  and  the  cumulative 


variance  reduction  are  listed  in  the  next  two  rows,  respectively.  The  component  loadings 


are  listed  below  their  respective  eigenvalue/variance  assessment. 
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Applying  Kaiser’s  approach  (retain  eignevalues  >  1)  to  the  data  in  Table  1 1 
indicates  the  dimensionality  of  the  data  is  two.  After  the  second  component,  all 
eignevalues  are  less  than  one.  The  first  component  provides  the  bulk  of  reduction  in 
variance.  Figure  8  graphically  depicts  Kaiser’s  criteria  for  dimensionality  and  the 


Figure  8,  Data  Dimensionality  By  Scree  Test  and  Kaiser  Criteria 

Catell’s  scree  test  for  the  data.  There  is  a  distinct  elbow  in  the  scree  test  plot  at  the 
second  eigenvalue  suggesting  a  data  dimensionality  of  two.  Retaining  the  two 
components  would  explain  over  78  percent  of  the  variance  in  the  data. 

4.2.3  Cluster  Analysis.  Cluster  analysis  was  apphed  as  a  tool  of  discovery  for 
insights  on  the  dimensionahty  of  the  response  data.  Hierarchical  clustering  techniques 
were  applied  to  all  the  responses  for  the  six  alternatives  via  the  centroid  linkage  method. 
The  centroid  method  is  an  agglomerative  approach  that  clusters  based  on  the  squared 
Euchdean  distance  between  the  means  of  two  clusters.  The  centroid  method  was  applied 
because  of  its  robustness  to  outliers  (IMP  Manual,  330:1995).  The  method  yielded  37 
clusters  for  the  responses  for  186  replications  (31  replications  of  six  alternatives).  There 
were  five  clusters  in  the  JMP  output  that  had  ten  or  more  clusters  assigned.  Although  the 
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majority  (greater  than  50%  of  the  replications)  of  the  F-15C  with  the  new  missile,  the  F- 
XX,  and  the  F-XX  with  the  new  missile  were  binned  in  three  separate  clusters,  the 
clustering  did  not  yield  any  trends. 

The  JMP  clustering  output  was  then  constrained  to  identify  six  clusters.  The 
threshold  of  six  was  chosen  commensurate  with  the  number  of  alternatives  under 
consideration.  The  number  of  replications  binned  in  each  cluster  is  depicted  in  Table  12. 
The  data  in  Table  12  indicates  the  replications  primarily  grouped  in  two  clusters.  This 
Table  12,  Cluster  Output  Constrained  to  Sbc  Clusters 


Cluster 

Members 

1 

116 

2 

62 

3 

2 

4 

1 

5 

4 

6 

1 

clustering  pattern  seems  to  support  the  dimensionality  insights  from  PCA.  Clustering 
data  is  in  Appendix  I.  The  data  indicated  two  clusters. 

4.2.4  FA.  Factors  were  extracted  from  the  correlation  matrk.  Varimax  rotation  was 
begun  on  two  factors  based  on  the  dimensionality  insights  from  the  PCA  and  clustering. 
Rotations  were  also  accomplished  for  three  and  four  factors.  The  results  of  the  two  and 
four  factor  rotations  are  in  Appendix  J.  The  loadings  after  three  factor  rotation  presented 
the  clearest  interpretation  of  the  factors.  The  structure  of  the  loadings  for  the  three  factor 
rotation  was  fairly  simple.  No  rows  had  more  than  one  high  correlation,  reducing  the 
complexity  and  ambiguity  of  the  interpretation.  The  loadings  from  the  three  factor 
varimax  rotation  are  depicted  in  Table  13.  The  significant  loadings  ( >  0.5)  are  listed  in 
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Table  13,  Rotated  Factor  Loadings 


RedLoss 

0.9545 

0.1617 

0.1510 

RedSort30 

-0.7955 

-0.3116 

-0.3081 

BLossSort 

-0.8442 

-0.2866 

-0.2897 

AirSup 

-0.8418 

-0.2856 

-0.2918 

BlueSA 

-0.4413 

-0.7069 

-0.1955 

BlueGrnd 

-0.2729 

-0.1738 

-0.9352 

-0.9061 

-0.0356 

-0.0727 

iRedFC 

0.0693 

0.9192 

0.0931 

bold  type.  RedLoss  loaded  positively  on  Factor  1.  Factor  I’s  remaining  significant 
loadings  (RedSortSO,  BLossSort,  AirSup,  RedOpen)  aU  loaded  negatively.  Factor  2  had  a 
significant  negative  loading  by  BlueS  A  and  a  positive  loading  by  RedFC.  The  third 
factor  had  a  significant  negative  loading  by  BlueGrnd.  Factor  interpretation  of  the  three 
factors  is  shown  in  Table  14.  The  strong  positive  loading  of  RedLoss  on  Factor  1  and  the 


Table  14,  Factor  Interpretations 


Factor 

Significant  Loadings 

Interpretation 

1 

RedLoss 

-RedSort30 

-BLossSort 

-AirSup 

-RedOpen 

Air-toAir  Lethality 

2 

-BlueSA 

RedFC 

SEAD  Effectiveness 

3 

-BlueGrnd 

IADS  Effectiveness 

negative  loads  of  RedSortSO,  BLossSort,  AirSup,  and  RedOpen  on  the  same  factor  give 
rise  to  the  factor  interpretation  of  Air-to-Air  Lethality.  This  captures  the  impact  of  the 
alternatives  on  Blue’s  overall  air-to-air  lethality.  The  strong  positive  loading  by  RedFC 
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and  moderately  high  negative  loading  of  BlueSA  support  the  interpretation  of  Factor  2  as 
describing  Suppression  of  Enemy  Defense  (SEAD)  Effectiveness.  The  factor  measures 
the  effect  of  the  alternatives  on  Blue’s  ability  to  negate  the  Red  SAM  threat.  The  high 
negative  loading  on  the  third  factor  leads  to  an  interpretation  of  Integrated  Air  Defense 
System  (IADS)  effectiveness.  This  factor  measures  the  impact  of  the  alternatives  on  the 
effectiveness  of  Blue  IADS  assets  (C^,  air-to-air  defensive  assets,  SAMs)  to  protect  Blue 
airbases. 

The  residual  matrix,  R  ,  provides  a  itKasure  of  the  fit  of  the  FA  to  the  data.  The 

-  RES 

R  for  the  varimax  rotated  factor  loadings  described  above  is  depicted  in  Table  15.  All 

-  RES 


Table  15,  Residual  Matrix 


Variables 

RedLoss 

RedSortSO 

BLossSort 

AirSup 

BlueSA 

BiueGrnd 

RedOpen 

RedFC 

RedLoss 

4.28E-06 

RedSortSO 

0.029 

3.55E-07 

BLossSort 

0.028 

-0.024 

4.84E-06 

AirSuD 

0.037 

0.035 

9.70E-04 

2.88E-06 

BlueSA 

0.014 

HEEBB 

■EEBB 

2.51  E-06 

BiueGrnd 

^^EEIQ 

■EE^ 

-2.72E-06 

RedOoen 

■EEES 

-0.11 

{■EEES 

^EESl 

0.058 

3.34E-07 

RedFC 

0.029 

0.033 

-1.53E-03 

0.022 

0.179 

^KE^ 

-1.96E-06 

cells  in  the  matrix  were  less  than  one  indicating  the  rotated  factors  fit  the  data  well.  The 
factor  scores  were  determined  from  the  factor  loadings  and  the  coefficients.  Factor 
scores  summarize  the  correlations  between  output  variables. 

Appendix  K  lists  the  scores  for  the  alternatives.  The  means  of  the  factor  scores 
for  each  alternative  are  listed  in  Table  16.  The  graphical  rendering  of  the  factor  scores 
provides  additional  insight  into  the  underlying  structure  of  the  data.  The  scores  were 
plotted  in  the  visualizations  depicting  output  responses  in  common  factor  space  discussed 
in  the  next  section. 
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Table  16,  Means  w/90%  Confidence  Intervals  of  Factor  Scores 


Alternatives 

Air-to-Air 

Lethality 

SEAD 

Effectiveness 

IADS 

Effectiveness 

F-15C 

F-15C  Msl 

F-15CRdr 

0.0046  ±0.2769 

F-15CRdr&Msl 

F-XX 

0.9654  ±  0.0904 

■iXiVJriiniMEM 

0.1075  ±0.2813  1 

F-XX  Msl 

4.3  Visualization 

Scatter  visualizations  were  used  to  visually  data  mine  the  responses  for  patterns 
and  anomalies  in  the  data.  The  scatter  visualization  tool  used  is  described  in  detail  in 
Chapter  6  of  the  MineSet  User’s  Manual  at  www.sgi.com  (MineSet  User’s  Manual, 
1997).  Buttons  on  the  panels  used  to  configure  the  visualizer  tool  discussed  in  this 
section  are  in  bold  type.  The  schema  used  for  the  visualization  in  the  thesis  effort  are  at 
Appendix  L.  The  data  files  used  in  the  visualization  were  Excel  spreadsheets  saved  in  a 
tab  delimited  format.  The  means  for  each  alternative  were  labeled  with  a  run  number  of 
“M”  in  the  data  files.  The  following  narrative  briefly  explains  the  configuration  of  the 
scatter  visualization  to  support  this  investigation. 

Figure  9  depicts  a  Data  Destination  Panel  setup  for  a  scatter  visualizer  session. 
The  button  on  the  data  destination  panel  at  the  top  left  labeled  scatter  visualizer 
indicates  the  MineSet  visualization  tool  selected.  The  user  specifies  the  data  fields  to  be 
visualized  in  the  area  labeled  visual  elements.  Only  those  selections  without  an  asterisk 
are  required  entries  to  launch  the  scatter  visualizer  tool  using  the  invoke  tool  button 
at  the  lower  right  corner  of  the  panel.  The  three  output  responses  mapped  to  the  axes  in 
the  figure  are  BLossSort,  BlueS  A,  and  RedFC.  The  size  of  the  entities  is  scaled  to  the 
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Figure  9,  Scatter  Visualizer  Data  Destination  Panel  (MineSet  User’s  Manual.  1997) 
RedLoss  response.  The  color  coding  of  the  data  entities  is  mapped  to  the  alternative 
solutions.  The  separate  data  entities  are  labeled  by  run  number. 

The  scatter  visualizer’ s  Options  Dialog  Box  activated  by  the  tool  options  button 
in  the  upper  right  comer  of  the  panel,  allows  the  user  to  label  axes  and  specify  entity 
color  mapping.  The  options  dialog  box  is  depicted  in  Figure  10.  The  axes  are  labeled, 
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Figure  10,  Scatter  Visualizer  Options  Dialog  Box  (MineSet  Users’  Manual.  1997) 
mapped  to  a  color,  and  sized  (as  required)  using  the  axes  input  areas  on  the  right 
side  of  the  dialog  box .  The  labels  listed  for  the  axis  labels  were  used  with  the  data 
destination  setup  just  described  in  Figure  10.  The  output  response  names  are  truncated  in 
the  figure.  The  upper  left  comer  of  options  dialog  box  has  spaces  for  the  user  to  map 
entities  to  colors,  shapes,  and  labels.  The  remaining  functions  in  the  box  were  not  used  in 
this  investigation. 

4.3.1  Output  Response  Visualization.  The  visualization  of  the  output  responses 
focused  on  three  areas  for  the  identification  of  patterns  and/or  anomalies:  output  response 
correlations,  outliers,  and  variability.  The  analyst  began  with  the  aggregate  data  view 
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(186  data  points  for  six  alternatives)  generated  by  the  scatter  visualizer.  The  view  of  the 
data  was  rotated  and  panned  to  provide  the  best  analyst  viewing  perspectives  for  the  data 
structure.  The  analyst  drilled  down  into  the  detail  of  the  data  at  the  aggregate  level  by 
selecting  the  display  of  summary  data  for  data  points  of  interest. 

The  Filter  Panel  depicted  in  Figure  1 1  was  used  to  drill  down  to  the  next  level  of 
output  data  by  selecting  specific  classes  of  data  points  for  display,  e.g.  the  F-XX 
replications  with  RedLoss  greater  than  300  and  BLossSort  less  than  0.0016.  The  filter 
was  also  used  to  drill  back  out  of  the  data  by  selecting  the  means  for  the  alternatives. 
Filter  choices  were  selected  by  highlighting  the  fields  of  interest  listed  in  the  three 
windows  at  the  top  of  the  figure  and  via  queries  using  the  buttons  and  input  boxes  in  the 
bottom  half  of  the  panel.  Means  of  the  output  responses  for  each  alternative  were  labeled 
with  an  “M”  in  the  run  field  of  the  data.  The  analyst  drilled  out  of  the  aggregate  data  and 
back  in  by  entering  “M”  in  the  input  box  adjacent  to  the  run  label.  The  entry  removes  all 
the  data  from  the  view  except  the  means.  Filtered  displays  were  rendered  with  the  scale 
to  filter  button  in  the  lower  right  comer  of  the  panel  in  the  “off’  position.  This  setting 
retained  the  scale  of  the  view  when  shifting  between  the  aggregate  and  filtered  viewing 
paradigms  to  facilitate  side-by-side  comparisons. 

The  scatter  visualization  of  the  output  responses  for  BlueSA,  RedFC,  and 
BlueGmd  was  noteworthy  as  the  one  case  of  all  displayed  where  the  improved 
performance  of  the  F-XX  and  F-XX  w/Missile  over  the  other  alternatives  was  not 
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Figure  11,  Scatter  Visualizer  Filter  Panel  (MineSet  User’s  Manual.  1997) 
apparent  from  the  aggregate  data  display.  The  underlying  structure  was  not  apparent 
until  the  data  was  filtered  to  display  only  the  means.  The  difficulty  in  discriminating  the 
performance  of  the  alternatives  is  explained  in  the  discussion  of  the  experimental  design 
analysis.  There  is  a  lack  of  statistical  significance  between  the  BlueSA  and  RedFC 
output  responses  for  several  alternatives. 

Outliers  in  the  data  were  detected  by  noting  those  points  appearing  to  have  a 
significant  separation  from  the  center  of  mass  for  each  alternative’s  data.  The  analyst 
zoomed  in  on  potential  outliers.  Display  of  the  summary  screen  provided  output  response 
information  for  the  point  of  interest.  The  aggregate  display  was  then  filtered  to  depict  the 
separation  between  the  point  and  the  mean  of  its  alternative  grouping  for  additional 
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insights.  The  twenty  replications  with  outliers  listed  in  standardized  matrix  of  Table  10 
were  displayed  and  visually  data  mined.  This  allowed  the  analyst  to  confirm  the  textual 
interpretation  that  there  are  no  patterns  in  the  outliers.  The  analyst  derived  insights  on  the 
variability  of  the  data  from  aggregate  views  and  views  of  the  output  response  means. 
Using  the  output  response  variances  on  the  diagonal  of  the  variance/covariance  matrix  in 
Table  8  enabled  the  analyst  to  focus  on  output  responses  with  extremes  of  variability. 
RedFC,  RedLoss,  and  RedOpen  were  visualized  for  insight  into  the  output  responses  with 
the  highest  variability.  The  MineSet  visual  display  for  RedFC,  RedLoss,  and  RedOpen  is 
depicted  in  Figure  12.  Although  the  MineSet  screen  display  does  not  lend  itself  well  to  a 
black  and  white  2D  rendering  in  the  figure,  the  dispersion  of  the  data  points  for  RedFC, 
RedLoss,  and  RedOpen  output  responses  depicted  in  Figure  12  confirms  the  significant 
variability  of  the  group,  particularly  the  RedFC.  BlueSA,  BLossSort,  and  RedSortSO 
were  visualized  for  insight  into  the  responses  with  the  lowest  variabihty.  The  MineSet 
visual  display  for  BlueSA,  BLossSort,  and  RedSortSO  is  depicted  in  Figure  13.  The 
compact  grouping  of  BlueSA,  BLossSort,  and  RedSort30  in  Figure  13  provides  insight 
small  variability  of  the  group. 

Overall,  the  response  output  visualization  did  not  yield  additional  insights  on  the 
structure  of  the  data  beyond  the  previous  data  analysis.  However,  the  capability  to  target 
interest  areas  and  drill  down  into  the  data  through  visualization  allowed  the  analyst  to 
come  to  quaUtative  insights  on  the  correlations,  outhers,  and  variability  of  the  data  more 
quickly  than  if  presented  a  table  of  simulation  output  to  assimilate.  The  visualization 
confirmed  analyst  conclusions  that  the  modifications  of  the  F-15C  radar,  replacement  of 
the  AMRAAM  with  a  new  missile,  and  RCS  reduction  to  implement  the  F-XX  yielded 
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Figure  12,  Scatter  Visualization  of  RedFC,  RedLoss,  and  RedOpen  Output  Responses 
progressively  improved  output  responses.  The  radar  improvements  and  addition  of  the 
new  missile  generally  yielded  the  same  performance.  The  exceptions  in  the  performance 
between  of  the  F-15C  with  radar  improvements  and  the  alternative  with  the  new  missile 
were  highlighted  by  the  visualization.  Otherwise,  the  F-15  configured  with  the  new 
missile  or  the  new  radar  yielded  the  same  performance  in  the  campaign  level  analysis. 
The  F-15C  configured  with  both  the  new  missile  and  radar  outperformed  either 
modification  (new  missile  or  radar)  alone  in  all  measures  except  BlueS  A  and  RedFC. 
However,  the  difference  in  the  performance  in  these  two  cases  does  not  exceed  the  90% 
confidence  level  half  lengths  for  the  response  outputs. 
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Figure  13,  Scatter  Visualization  of  BlueSA,  BLossSort,  and  RedSortSO  Output  Responses 


4.3.2  Factor  Score  Visualization.  After  exploring  the  responses,  the  factors 
were  visualized.  The  three  axes  of  the  scatter  visualization  were  labeled  Air-to- Air 
Lethality,  SEAD  Effectiveness,  and  IADS  Effectiveness.  The  size  of  the  entities  and 
entity  labels  were  based  on  notional  LCC  as  follows: 


F-15C  $29M 

F-15C  with  new  missile  $33M 

F- 1 5C  with  new  radar  $35M 

F-  15C  with  new  radar  and  missile  $40M 
F-XX  $70M 

F-XX  with  new  missile  $77M 
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Entity  colors  were  mapped  to  the  same  fields  as  in  the  output  response  visualizations.  A 
measure  of  the  adequacy  of  factor  rotation  is  the  orientation  of  clusters  of  factor  scores 
relative  to  the  axes.  The  visualization  of  the  aggregate  data  indicated  clustering  of  the 
data  points  around  the  air-to-air  lethality  axis.  However,  the  clustering  is  not  definitive 
enough  to  make  an  assessment  of  the  adequacy  of  the  rotation. 

The  aggregate  factor  scores  are  fairly  widely  dispersed  in  the  MineSet  3D 
visualization  of  the  data.  The  variability  in  the  factor  scores  of  the  F-15C  alternatives 
(radar  improvement,  new  missile,  both)  is  apparent  from  the  visualization.  It  is  difficult 
from  the  aggregate  data  to  quantify  one  of  the  F-15C  options  (radar,  new  missile,  both)  as 
better  than  another  except  that  aU  show  improvement  over  the  current  air  superiority 
fighter.  The  data  points  for  the  F-XX  and  F-XX  w/new  missile  are  more  tightly  grouped 
indicating  less  variability.  The  F-XX  and  F-XX  w/new  missile  factor  scores  are  clearly 
separated  from  the  F-15C  alternatives.  Additionally,  the  center  of  mass  for  F-XX  w/new 
missile  factor  scores  is  clearly  to  the  right  of  the  F-XX  suggesting  a  synergistic  effect 
between  the  new  missile  and  the  reduced  RCS  that  did  not  exist  with  the  radar 
improvement  alone.  The  visual  display  of  the  aggregate  factor  scores  show  the  center  of 
mass  for  each  alternative  (F-15C,  F-15C  w/missile,  F-15C  w/radar,  F-15C  w/radar  and 
missile,  F-XX,  F-XX  w/missile)  progressively  transiting  the  graphic  to  a  quadrant  of 
positive  loading  in  air-to-air  lethality,  DCA  effectiveness,  and  SEAD  effectiveness  factor 
scores. 

Filtering  the  factors  scores  to  display  the  means  for  each  alternative  indicates  the 
F-15C  w/missile  shows  greater  improvement  over  the  F-15C  than  the  radar  improvement. 
The  area  of  greatest  improvement  over  the  base  cases  for  all  alternatives  is  in  the  air-to- 
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air  lethality  factor.  This  finding  tracks  with  the  large  amount  of  variability  in  the  data 
explained  by  the  first  component  in  the  PCA.  IADS  and  SEAD  effectiveness  show 
moderate  to  low  improvement  between  the  F-15C  and  the  modifications  (new  missile, 
radar,  both).  Additionally,  the  F-15C  modifications  and  the  F-XX  vary  little  in  DCA 
effectiveness.  The  F-XX  w/new  missile  shows  significant  increase  over  the  other 
alternatives  in  DCA  effectiveness.  Likewise,  the  F-15Cs  and  the  F-XX  had 
approximately  the  same  level  of  improvement  for  SEAD  effectiveness  except  a 
decreasing  effectiveness  by  the  F-15C  w/new  missile  and  radar.  This  anomaly  may  be 
due  to  the  statistical  significance  issues  for  BlueSA  and  RedFC  discussed  previously  in 
this  section.  The  F-XX  and  F-XX  w/new  missile  show  improvements  in  all  areas. 

The  visualization  of  the  mean  factor  scores  in  Figure  14  depicts  a  side  view  of  the 
cost/effectiveness  integration  plot  for  the  campaign  analysis.  Figure  15  depicts  the  top 
view  of  the  plot.  The  operational  effectiveness  of  the  six  alternatives  are  their  air-to-air 
lethality,  SEAD  effectiveness,  and  IADS  effectiveness  mean  factor  scores  represented  in 
three  space.  The  notional  LCCs,  in  millions  of  dollars,  is  represented  by  the  size  and 
labels  on  the  entities.  Both  figures  underscore  that  the  average  improvement  in  the  air-to- 
air  lethality  performance  of  the  F-15C  w/Radar  &  Missile,  F-XX,  and  F-XX  w/Missile  is 
greater  than  the  average  improvement  for  said  alternatives  in  the  SEAD  or  IADS  labels 
on  the  entities.  Both  figures  underscore  that  the  average  improvement  in  the  air-to-air 
lethality  performance  of  the  F-15C  w/Radar  &  Missile,  F-XX,  and  F-XX  w/Missile  is 
greater  than  the  average  improvement  for  said  alternatives  in  the  SEAD  or  IADS 
effectiveness  performance.  The  two  plots  also  graphically  display  the  average  overall 
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Figure  14,  Scatter  Visualization  of  Mean  Factor  Scores  -  Side  View 
operational  effectiveness  (air-to-air  lethality,  SEAD  effectiveness,  IADS  effectiveness)  of 
the  F-XX  w/Missile  is  greater  than  the  other  alternative  solutions 
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Figure  15,  Scatter  Visualization  of  Mean  Factor  Scores  -  Top  View 
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5.  Conclusions  and  Recommendations 


This  chapter  discusses  the  conclusions  and  recommendations  drawn  from  the 
analysis. 

5.1  Summary 

This  thesis  effort  developed  an  approach  to  reduce  the  dimensionality  of  multiple 
campaign  level  measures  of  effectiveness  for  an  AoA.  Additionally,  an  AoA 
visualization  paradigm  was  developed  for  the  capability  demonstration  of  ASC’s  SIMAF. 

The  campaign  level  analysis  and  AoA  results  discussed  in  the  thesis  were  based 
on  a  notional  acquisition  of  a  new  air  superiority  fighter,  the  F-XX.  The  on-going 
acquisition  and  fielding  of  the  F-22  fighter  was  not  addressed  in  the  study.  An 
experimental  design  with  two  treatments,  one  with  three  levels  of  F-15C  and  one  with 
two  levels  of  weapons  configuration,  was  the  basis  of  the  campaign  level  simulation 
performed.  The  F-XX  is  characterized  as  the  level  of  F-15C  with  an  improved  radar  and 
reduced  RCS.  The  DOE  yielded  six  design  points;  the  F-15C,  the  F-15C  w/Missile,  the 
F-15C  w/Radar,  the  F-15C  w/Radar  &  Missile,  the  F-XX,  and  the  F-XX  w/Missile. 

AFSAA’s  THUNDER  6.4.2  campaign  level  model  was  used  in  the  simulation. 
Unclassified  THUNDER  data  files  for  a  SWA  scenario  provided  by  ASC/XR  were  used 
in  the  model.  Each  alternative  was  assessed  for  its  contribution  to  the  campaign  objective 
of  gaining  and  maintaining  air  superiority.  The  ten  simulation  output  responses  collected 
to  measure  the  accomplishment  of  the  campaign  objective  were  chosen  based  on 
recommendations  in  an  ASPVG  report  discussing  THUNDER  use  in  campaign  analysis. 
Thirty  one  replications  of  a  30  day  war  were  performed  at  each  design  point.  Output 
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responses  for  the  F-15C,  F-15C  w/Missile,  F-15C  w/Radar,  and  F-15C  w/Radar  & 

Missile  alternatives  necessitated  the  use  of  Bonferroni  confidence  intervals  to  make 
simultaneous  comparisons.  The  comparisons  indicated  the  F-15C  w/Missile  design  is 
generally  better  than  F-15C  (improved  performance  in  six  out  of  ten  output  responses), 
the  F-15C  w/Radar  design  is  generally  better  than  the  F-15C  (improved  performance  in 
two  out  of  ten  output  responses),  the  F-15C  w/Radar  &  Missile  design  is  generally  better 
than  the  F-15C  (improved  performance  in  six  out  of  ten  output  responses),  the  F-15C 
w/Radar  &  Missile  design  is  generally  better  both  the  F-15C  w/Missile  and  F-15C 
w/Radar  (improved  performance  in  three  out  of  ten  output  responses  for  each  alternative). 

A  Wilcoxon  signed  rank  test  was  used  to  compare  the  distributions  of  the  output 
responses  for  the  F-15C  and  the  F-XX  w/  Missile.  Two  output  response  fields  with 
identical  distributions  for  the  two  alternatives  were  not  included  in  the  multivariate 
analysis. 

Multivariate  and  visualization  tools  were  used  to  reduce  the  dimensionality  of  the 
AoA  operational  effectiveness  measures  -  the  simulations  output  responses.  Three 
multivariate  data  reduction  techniques  were  used  to  assess  the  structure  of  the  data;  PCA, 
cluster  analysis,  and  FA.  Although  PCA  and  clustering  suggested  an  underlying  data 
structure  with  a  dimensionality  of  two,  the  FA  yielded  the  clearest  interpretation  after  an 
orthogonal  (varimax)  rotation  of  three  factors.  The  loadings  of  the  rotated  variables 
indicated  an  interpretation  of  the  three  factors  as  Air-to-Air  Lethality,  SEAD 
Effectiveness,  and  IADS  Effectiveness. 
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The  results  of  an  AoA  are  presented  in  terms  of  a  comparison  of  the  LCC  versus 


the  operational  effectiveness  of  the  alternatives.  One  approach  to  presenting  an  AoA 
cost-effectiveness  assessment  for  the  six  alternatives  is  the  3D  bar  chart  depicted  in 
Figure  16.  The  mean  factor  scores,  representing  a  three  dimensional  assessment  of  each 
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Figure  16,  Cost/Effectiveness  Comparison  Bar  Chart 
alternative’s  operational  effectiveness,  are  indicated  by  the  bars  in  the  figure.  The  scores 
have  been  transformed  to  the  positive  axes.  Notional  LCCs  of  the  alternatives  in  millions 
of  dollars  are  immediately  to  the  right  of  the  alternative  labels.  The  factor  interpretations 
are  at  the  bottom  of  the  figure.  The  figure  indicates  that  the  greatest  area  of  improvement 
in  operational  effectiveness  of  the  F-15C  w/Radar  &  Missile,  F-XX,  and  F-XX  w/Missile 
over  the  current  air  superiority  fighter  is  average  air-to-air  lethality  performance.  The 
alternatives  generally  show  improvement  in  average  SEAD  and  IADS  Effectiveness 
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performance  over  the  F-15C.  However,  average  SEAD  and  IADS  effectiveness  between 
alternatives  is  not  significantly  different  except  for  the  F-XX  w/Missile.  Figure  16  alone 
does  not  provide  a  definitive  answer  to  the  question  of  F-XX  acquisition.  In  acquisition 
deliberations,  the  decision  maker  would  consider  the  cost/effectiveness  information 
provided  by  the  figure  in  light  of  budget,  force  structure,  schedule,  etc. 

Visualization  of  the  data  via  high-end  visualization  software  and  a  workstation  for 
high  resolution  graphics  and  animations  provided  additional  insights  on  the  underlying 
structure  of  the  data.  The  simulation  output  responses  and  multivariate  analysis  results 
were  preprocessed  using  IMP  data  mining  algorithms.  Subsequently,  the  IMP  output  was 
visually  data  mined  by  rendering  a  graphic  of  the  data  and  interacting  with  the  3D 
visualization.  User/visualization  interface  was  through  use  of  the  mouse  to  select  entities 
of  interest  and  user  developed  queries. 

The  output  responses  and  factors  scores  were  displayed  using  the  SGFs  MineSet 
visualization  tool.  The  visualization  confirmed  the  relationships  between  the  output 
responses  and  highlighted  the  variability  differences  between  F-15C  alternatives  with  and 
without  the  reduced  RCS.  The  factor  analysis  visualization  provided  the  similar  insights 
into  the  effects  of  RCS  on  variability. 

The  differences  in  variability  between  the  reduced  RCS  and  the  other  alternatives 
is  due  to  the  effect  of  RCS  on  ground  controlled  intercept  (GCI)/AEW  detection  ranges 
and  the  probability  of  detection  by  threat  aircraft.  The  reduced  RCS  was  implemented 
via  an  RCS  value  in  the  typeac.dat  file  and  reduced  detection  probabilities  in  the 
detect.dat  file  for  the  F-XX.  Consequently,  it  will  be  less  likely  that  there  will  be 
detections/  engagements  of  the  F-XX  by  adversary  interceptor  aircraft.  The  reduction  of 
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stochastic  engagement  events  by  adversary  aircraft  reduces  the  variability  of  the  output 
responses  for  the  F-XX  scenarios. 

The  factor  scores  displayed  in  three  space  accentuated  the  significant 
improvements  between  the  alternatives  in  air-to-air  lethality  and  the  marginal 
improvements  in  SEAD  effectiveness  and  IADS  effectiveness.  The  F-XX  w/Missile 
indicated  an  improved  SEAD  and  IADS  effectiveness  performance  over  other 
alternatives. 

The  value  added  by  data  mining  via  visualization  of  the  data  was  the  faster 
information  assimilation  through  user  interaction  with  the  data. 


5.2  Findings 

This  investigation  yielded  the  following  findings  from  the  simulation  responses 
for  the  notional  F-XX  AoA  study  scenario  used  in  this  thesis  effort: 

1)  The  F-15C  w/Missile  showed  improved  RedLoss,  RedSortSO,  BLossSort, 
AirSup,  RedFC,  and  RedTEL  performance  over  the  F-15C. 

1)  The  F-15C  w/Radar  showed  improved  RedSort30  and  AirSup  performance 
over  the  F-15C. 

3)  The  F-15  w/Radar  &  Missile  showed  improved  RedLoss,  RedSort30, 
BLossSort,  AirSup,  BlueGrnd,  and  RedOpen  performance  over  the  F-15C. 

4)  The  F-15C  w/Radar  &  Missile  showed  improved  RedLoss,  AirSup,  and 
RedOpen  performance  over  the  F-15C. 

5)  The  F-15C  w/Radar  &  Missile  showed  improved  RedLoss,  AirSup,  and 
RedOpen  performance  over  the  F- 1 5C. 

6)  RedAQ  and  RedTEL  output  responses  were  not  statistically  significant  for  a 
comparison  of  the  F- 1 5C  and  F-XX  w/Missile. 

7)  The  eight  measures  of  effectiveness  for  the  AoA  can  be  represented  by  three 
interpretable  factors  through  the  application  of  FA  -  Air- to- Air  Lethality,  SEAD 
Effectiveness,  and  IADS  Effectiveness. 
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8)  3D  visualization  of  the  mean  factor  scores  for  the  six  alternatives  indicated  the 
greatest  improvement  for  the  three  factors  was  in  the  average  air-to-air  lethality  factor 
scores  for  the  F-15C  w/Radar  &  Missile,  F-XX,  and  F-XX  w/Missile. 

5.3  Lessons  Learned 

The  THUNDER  basic  course  presented  by  the  model’s  vender,  831  Inc., 
significantly  improved  the  user’s  learning  curve  in  running  the  simulation.  The  course 
covered  fundamentals  of  data  file  manipulation  and  air  war  planning.  Students  who 
undertake  future  THUNDER-based  theses  should  consider  taking  the  THUNDER  basic 
course  as  funds  and  time  permit.  Other  excellent  THUNDER  resources  used  in  this  effort 
were  the  on-line  THUNDER  Analyst  Manual  and  advanced  THUNDER  analyst  course 
material  available  at  the  C3I  website,  www.c3I.com.  The  documentation  was  particularly 
useful  for  the  analysis  of  the  simulation  output. 

Access  to  a  CD  disk  writer  was  useful.  The  THUNDER  output  (graphics  and 
textual  files)  for  this  investigation  exceeded  4()0M  for  each  31  replication  run  of  the  30 
day  war.  The  CD  writer  allowed  the  user  to  archive  the  data  from  each  run  on  a  CD 
(650M  capacity).  This  capability  also  helped  manage  limited  hard  disk  storage  space  on 
the  workstation  dedicated  to  the  investigation. 

Analysis  of  the  simulation  output  was  greatly  streamlined  through  the  use  of  the 
UNIX  script  ttgraph.nawk  and  the  use  of  EXCEL  macros.  The  UNIX  script  allowed  the 
user  to  convert  the  THUNDER’S  ttgraph.rpt  files  to  comma  delimited  files  that  could  be 
imported  into  EXCEL.  The  use  and  coding  of  the  ttgraph.nawk  script  is  described  in 
Grier’s  thesis  (Grier,  1997:107).  The  EXCEL  macros  were  used  to  populate  spreadsheets 
for  each  alternative  from  the  comma  delimited  files,  extract  data  from  the  data  set  of  each 
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replication’s  output  data,  and  build  tables  consolidating  the  data  from  all  the  replications 
for  each  alternative. 

Once  the  visualization  tool  for  the  effort  was  chosen,  the  MineSet  operators’ 
manual  on  the  SGI  website  was  a  great  learning  aid  to  utilize  the  software.  The  manual 
presented  examples  of  the  schema  required  to  configure  MineSet  visualizer  tools  and 
discussed  the  use  of  input  screens  to  customize  and  interact  with  the  visualizations.  The 
MineSet  team  at  SGI  provided  quick  and  detailed  feedback  to  my  e-mailed  questions. 
While  there  are  other  commercially  available  visualization  and  data  mining  packages, 
MineSet’s  ease  of  use  for  an  analyst  with  limited  UNIX  and  database  knowledge  was 
noteworthy. 

The  only  drawback  I  found  with  the  use  of  the  MineSet  tool  was  in  converting 
screen  graphic  images  to  black  and  white  images  for  publication  in  this  thesis.  MineSet 
allows  the  user  to  save  screen  images  in  an  .rgb  format.  The  .rgb  format  files  can  be 
saved  to  .gif  files  using  an  IRIX  utility  called  tog/jf  available  at  the  SGI  website. 
Microsoft’s  Photo  Shop  utility  can  be  used  to  reverse  the  black  background  in  the 
MineSet  scatter  visualization  images  and  enhance  the  rendering  of  the  entities,  but  the 
publication  quality  is  still  limited. 

This  effort  required  the  use  of  several  software  packages  for  visualization  design 
(MineSet)  and  presentation  (Hummingbird  EXCEED  Multiplatform  and  3D).  Although 
funding  was  available  to  purchase  the  software,  often  the  government  acquisition 
procedures  delayed  delivery.  Fortunately,  the  software  used  in  this  effort  was  available 
in  fully  operational  demo  copies  at  websites  and  through  sale  representatives.  ENS’s  host 
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administrator  for  the  modeling  and  simulation  department  was  pivotal  in  assisting  with 
the  procurement,  installation,  and  operation  of  the  software. 

Each  set  of  an  alternative’s  thirty  one  replications  of  the  30  day  war  on 
THUNDER  for  this  effort  took  over  sixteen  and  a  half  hours  to  complete  using  the 
simulation’s  ttrep  script.  Thesis  research  by  Davies  during  the  same  timeframe  of  this 
effort  utilized  a  script  to  run  multiple  THUNDER  replications  on  numerous  central 
processing  units  (CPU)  at  ASC’s  Major  Shared  Resource  Center  (MSRC) 
simultaneously.  Davies’  approach  drastically  reduced  the  time  to  make  multiple 
THUNDER  replications  (Davies,  1998).  He  was  able  to  complete  thirty  one  replications 
of  a  34  day  war  in  less  than  two  hours.  Future  THUNDER  theses  efforts  should  consider 
using  said  script  on  MSRC  or  AFIT  UNIX  network  assets. 

5.4  Recommendations  for  Future  Study 

Data  mining  has  significant  applications  to  operations  research.  Patterns  and 
classifications  of  large  sets  of  data,  such  as  simulation  output,  that  can  be  revealed/made 
through  data  mining  are  clearly  of  benefit  to  the  military  analyst.  There  are  numerous 
data  mining  tools  available  commercially  including  statistic  packages  such  as  JMP.  The 
MineSet  data  mining  software  utilized  in  this  investigation  was  exclusively  used  for  its 
visualization  capability.  The  extensive  capability  of  the  software  to  apply  data  mining 
algorithms  to  large  data  sets  through  the  use  of  the  client/server  module  was  not 
investigated. 

Future  investigations  could  use  MineSet’s  data  mining  capabilities  to  drill  into 
large  data  sets  for  pattern  recognition  and  classification.  Additionally,  thesis  efforts 
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could  investigate  dependencies  within  simulation  data  sets  via  MineSet’s  decision  tree 
functions. 
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Appendix  A 


The  following  outline  is  the  Suggested  Analysis  of  Alternatives  (AoA)  Report  format 
provided  by  the  Air  Force  Center  of  Expertise  for  AoAs,  the  Office  of  Aerospace  Studies 
(Draft  OASP  97-1,  1997:30).  In  an  actual  AoA  study,  the  campaign  level  analysis 
discussed  in  the  thesis  would  be  documented  in  section  4  of  the  suggested  format  below. 

TITLE 

EXECUTIVE  SUMMARY 

1.  INTRODUCTION 

1.1.  Background 

1.2.  Purpose 

1.3.  Scope 

2.  ACQUISITION  ISSUES 

2.1.  Mission  Need 

2.2.  Threat(s) 

2.3.  Scenarios 

2.4.  Environment 

2.5.  Constraints  and  Assumptions 

2.6.  Operations  Concept 

3.  ALTERNATIVES 

3.1.  Description  of  Alternatives 

3.2.  Nonviable  Alternatives 

4.  ANALYSIS  OF  ALTERNATIVES 

4.1.  Methodology 

4.1.1.  Effectiveness  Methodology 

4.1.2.  Cost  Methodology 

4.1.3.  Cost-Effectiveness  Methodology 

4.2.  Models  and  Data 

4.2.1.  Effectiveness  Models  and  Data 

4.2.2.  Cost  Models  and  Data 

4.2.3.  Cost-Effectiveness  Models  and  Data 

4.3.  Effectiveness  Analysis 

4.3.1.  Mission  Tasks  (MTs) 

4.3.2.  Measures  of  Effectiveness  (MOEs) 

4.3.3.  Measures  of  Performance  (MOPs) 

4.3.4.  Effectiveness  Sensitivity  Analysis 

4.3.5.  TEMP/ORD/AoA  Linkage 

4.4.  Life  Cycle  Cost  Analysis 

4.4.1.  Research  and  Development  (R«&D)  Cost 

4.4.2.  Investment  Cost 

4.4.3.  Operations  and  Support  (O&S)  Cost 

4.4.4.  Disposal  Cost 

4.4.5.  Cost  Sensitivity  Analysis 

4.5.  Cost-Effectiveness  Analysis 

4.6.  Tradeoff  Analysis 


86 


4.7.  Ranking  and  Decision  Criteria 
5.  SUMMARY  OF  RESULTS 

5.1.  Results  and  Conclusions 

5.2.  Recommendations 

ATTACHMENT  A:  OPERATIONAL  EFFECTIVENESS  SUPPORTING 
ANALYSIS/DOCUMENTATION 

ATTACHMENT  B:  COST  SUPPORTING  ANALYSIS/DOCUMENTATION 
ATTACHMENT  C:  OTHER  SUPPORTING  ANALYSES/DOCUMENTATION 
ATTACHMENT  D;  OTHER  SUPPORTING  DOCUMENTATION 
ATTACHMENT  E.  RESPONSIBLE  TEAM  MEMBERS  AND  ORGANIZATIONS 
ATTACHMENT  F;  ACRONYMS 
ATTACHMENT  G.  REFERENCES 


87 


Appendix  B 


This  table  lists  the  operational  objectives  and  operational  tasks  under  the  Gain  and 
Maintain  Air  Superiority  campaign  objective  that  can  be  measured  by  THUNDER 
(ASPVG,  1995:4).  Additionally,  the  mapping  of  the  output  identifiers  for  the  metrics 
measured  in  the  unclassified  data  base  to  the  operational  tasks  are  listed. 


Operational  Objectives 

Operational  Tasks 

Metric 

Defeat  air  attacks 

Destroy/disrupt  aircraft  and  helicopters  in 
flight. 

RedLoss 

BLossSort 

Destroy/disrupt  cruise  missiles  in  flight 

N/A 

Disrupt  sensors  on  aircraft  and  weapons 

N/A 

Execute  passive  defense  measures  in 
threatened  areas 

BlueGmd 

Suppress  generation  of  air 
sorties 

Crater/mine/damage  airfield  runways  and 
taxiways 

ISMil 

Destroy/damage  aircraft  in  the  open  or  in 
revetments 

RedOpen 

RedSortSO 

AirSup 

Destroy/damage  aircraft  in  hardened  shelters 

WSSSi 

Destroy/damage  airbase  support  facilities 

ilMI 

Deny  attack  helicopter  forward  area 
refuel/replenishment  points  (FARRP) 

N/A 

Suppress  surface-based  air 
defenses 

Destroy/damage  fixed  surface-to-air  missile 
(SAM)  launchers 

RedTEL 

RedACQ 

RedFC 

BlueSA 

Destroy/damage  SAM  launchers  and  anti¬ 
aircraft  artillery  (AAA) 

RedTEL 

RedACQ 

RedFC 

BlueSA 

Destroy/disrupt  tracking  and  engagement 
radars 

N/A 

Defeat  attacking  ballistic 
missiles 

Destroy  ballistic  missiles  in  flight  (active 
defense) 

N/A 

Execute  passive  defense  measures  in 
threatened  areas 

N/A 
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Operational  Objectives 

Operational  Tasks 

Metric 

Suppress  the  generation  of 
ballistic  missile  launches 

Damage/destroy  transportable  erector 
launchers  (TELs)  in  the  field  and  disrupt 
operations 

N/A 

Damage/destroy  TELs  in  garrisons  and 
assembly  areas 

N/A 

Damage/destroy  fixed  TBM  launchers 

N/A 
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Appendix  C 


This  appendix  alphabetically  lists  the  abbreviations  used  for  the  THUNDER  output 
metrics. 

AirSup  Days  to  air  supremacy  (defined  as  10%  of  initial  Red  sortie 

generation  rate) 

BLossSort  Blue  air-to-air  loss  per  blue  sortie  flown 

BlueGmd  Total  number  of  Blue  aircraft  destroyed  on  the  ground 

BlueS  A  Number  of  Blue  aircraft  lost  to  enemy  surface-to-air  missile  threats 

per  sortie 

RedACQ  Number  of  acquisition  radars  killed 

RedFC  Number  of  fire  control  radars  killed 

RedLoss  Red  aircraft  lost  due  to  Blue  air 

RedOpen  Number  of  Red  aircraft  destroyed  in  the  open 

RedSortSO  Red  sortie  generation  capability  on  day  30 

RedTEL  Number  of  transportable  erector/launchers  killed 
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Appendix  D 


This  appendix  contains  excerpts  of  the  modified  THUNDER  data  files  for  each  AoA 
alternative  solution.  The  files  are  listed  in  the  order  they  are  discussed  in  the  text.  Some 
files  are  included  in  their  entirety. 

1’he  changes  to  the  files  from  the  original  data  files  provided  by  ASC/XR  are  shaded  and 
itatidzeii  with  notations  in  bold  type, 

control.dat  -  All  scenarios 

CONTROLS. 020 

REPORT. TITLE  "UNCLASSIFIED  THUNDERSTORM  " 

BAS IC . S IMULAT ION . CONTROLS 

mmER,  OF,  DAYSr  OF,  WAR  (IMT,  DAYS) 

. 'begin 'daylight  . 

END . DAYLIGHT (DEC . HOURS) 

ADX . PROVIDE . TERM . AD . FOR .FIXED . TGTS 
IGNORE . RANGE . CONSTRAINTS 
COUNT . ALERT . MSN . AS . COMPLETE 
END . BASIC . SIMULATION . CONTROLS 

GAME. CONTROLS 

GAME. FILE. MOD. FLAG 
GAME. ENABLE 
GAME. CYCLE 

FIRST. GAME. STOP (DAYS) 

END . GAME . CONTROLS 

ADF . CALIBRATION . CONTROLS 

ADF. CALIBRATION. MODE. ENABLE  NO 

@  if  enabled 

@  only  preplanned  ATOs  fly 

@  all  ADvsAC  Pks  =0.0 

@  unlimited  AD  ammo  reloads 

0  lethal  SEAD  disabled 

0  air-to-ground  disabled 

0  special  ADF  calibration  transactions  enabled 

0  ADF  results  averaged  over  multiple  reps: 


REPLICATIONS . PER . ADF . CALCULATION  3 

USE . MANUAL . IADS . IADS . VALUES (YES, NO)  NO 

MANUAL . IADS . INTEGRATION. LEVEL (0 . 0-1 .0)  0.7 

MANUAL. IADS. SECONDS. DELAY  15.0 

MANUAL. IADS. INTIMIDATE  0.0 

END . ADF . CALIBRATION . CONTROLS 

COMPUTATIONAL . RESOLUTION . LEVELS (LOW, HIGH) 

BLUE. AIR. DEFENSE  HIGH 

RED. AIR. DEFENSE  .  HIGH 

AIR. WAR  HIGH 

BLUE . RECONNAISSANCE  HIGH 

RED . RECONNAISSANCE  HIGH 

WEATHER  LOW 

GROUND. MOVEMENT  HIGH 


YES 

NO 

24 

1.1 
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CRITICAL. RESOURCES  LOW 

END . COMPUTATIONAL . RESOLUTION . LEVELS 

GRAPHIC . OUTPUT . CONTROLS 

WRITE . GRAPHIC . TO . TRANS  NO 

POST . PROCESS . GRAPHICS ( BOTH , CHART , MAP , NONE )  BOTH 
AD . COMPLEX . STATUS . UPDATE . CYCLE (HOURS)  12 

AIRBASE . STATUS .UPDATE .CYCLE (HOURS)  12 

CHOKE .POINT . STATUS .UPDATE .CYCLE (HOURS)  12 

COMMAND . STATUS .UPDATE .CYCLE (HOURS)  12 

LOG . FAC . STATUS . UPDATE . CYCLE ( HOURS )  1 2 

STRATEGIC . TARGET . STATUS . UPDATE .CYCLE (HOURS)  12 
SUPPLY . TRAIN . STATUS .UPDATE . CYCLE (HOURS)  12 

TBM. UNIT. STATUS. UPDATE. CYCLE (HOURS)  12 

UNIT . STATUS . UPDATE . CYCLE (HOURS)  12 

PERCEPT . STATUS . UPDATE . CYCLE (HOURS)  12 

AIR . REPORT . CYCLE (DAY, GAME, BLUE, RED)  DAY 

END . OUTPUT . CONTROLS 


OVERALL . REPORT . CONTROLS 


REPORT. MODE (STANDARD, METRIC, 
END . OVERALL . REPORT . CONTROLS 

BOTH) 

STANDARD 

DATA . REPORT . CONTROLS 

NUMBER . OF . DATA . REPORTS 

73 

CONTROL. FILE 

DR-2 

NO 

RANDOM. SEEDS 

DR-3 

NO 

STANDARD . TARGETS 

DR-5 

NO 

TYPE . RADAR 

DR-10 

NO 

TYPE . JAMMER 

DR- 11 

NO 

AIR . DEFENSE . DATA . REPORTS 

**** 

TYPE. AIR. DEFENSE 

DR-12 

NO 

POSTURES . & . CLASSES 

DR- 13 

NO 

INTEGRATED . AD . SYSTEM 

DR- 14 

NO 

ADA . ENGAGEMENT . PROBS 

DR- 15 

NO 

AD . VS . AIRCRAFT . PKS 

DR-27 

NO 

AD. COMPLEXES 

DR- 6  6 

NO 

PLANNING . DATA . REPORTS 

*  *  *  * 

AIR. RULES 

DR-20 

NO 

AIR. ALLOCATION 

DR-21 

NO 

AIR . PLANNING . FACTORS 

DR- 67 

NO 

OCA. TARGETS 

DR-23 

NO 

PREPLANNED. ATOS 

DR-29 

NO 

ZONE . SECTOR . TGT . PRIORITY 

DR- 65 

NO 

INT . DEPTH . FACTOR . CURVES 

DR- 68 

NO 

ST I. TARGETS 

DR- 70 

NO 

AIR . TO . AIR . DATA . REPORTS 

**** 

AIR. TO. AIR. PKS 

DR-24 

NO 

RELEASE . RANGE . ADVANTAGE 

DR-25 

NO 

CONFIG . DETECT . PROBS 

DR-26 

NO 

AIR. TO. GROUND. PKS 

DR-28 

NO 

HARM. PKS 

DR- 40 

NO 

MINE. PKS 

DR- 41 

NO 

TAKEOFF . LANDING . PKS 

DR- 4  2 

NO 

TYPE . AIR . MUNI T IONS 

DR-31 

NO 

TYPE. AIRCRAFT 

DR-32 

NO 

AIRCRAFT .MAINTENANCE 

DR-39 

NO 
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TBM. UNITS 

DR-75 

NO 

TBM. DETECT ION 

DR-76 

NO 

AIRBASE 

DR-33 

NO 

CARRIER . BATTLE . GROUP 

DR- 30 

NO 

SUPPLY. KIT 

DR-34 

NO 

SQUADRON 

DR-35 

NO 

AIR.  TO. GRD. MIN. CEIL. VIS 

DR-36 

NO 

SUPPLY . URGENCY . CURVES 

DR- 3  7 

NO 

CRITICAL. RE SOURCES 

DR- 3  8 

NO 

NODES 

DR-45 

NO 

TRANS . SHIPMENT . POINTS 

DR-46 

NO 

ARCS 

DR- 4  7 

NO 

CHOKE. POINTS 

DR-48 

NO 

MINE . DELAY . CURVES 

DR- 4  9 

NO 

BATTLEFIELD 

DR-50 

NO 

AIR. NETWORK 

DR-73 

NO 

GROUND . TARGET .ACQUISITION 

DR-51 

NO 

TERRAIN 

DR-52 

NO 

INTERVISIBILITY . CURVES 

DR-53 

NO 

TYPE. C3. FACILITY 

DR-55 

NO 

COMMAND 

DR- 5  6 

NO 

GROUND . RULES 

DR-57 

NO 

SUPPLY . TRAIN . TARGETS 

DR-54 

NO 

LOGISTIC . FACILITY 

DR-58 

NO 

TYPE. UNIT 

DR-59 

NO 

TYPE . EQUIPMENT 

DR- 60 

NO 

EQUIPMENT . SIZE . CATEGORY 

DR- 61 

NO 

WEAPON . VS . EQUIPMENT . EFFECTS 

DR- 62 

NO 

UNIT 

DR-63 

NO 

UNIT . SUPPLY . CURVES 

DR- 81 

NO 

LIFT. EVENTS 

DR-82 

NO 

PERCEPTIONS 

DR- 6  4 

NO 

STRATEGIC . TARGETS 

DR- 6  9 

NO 

AIRCRAFT . FACTORS 

DR-71 

NO 

SREC .MAX. EFFECT .MULTIPLIER 

DR-72 

NO 

BOOST . PHASE . INTERCEPT 

DR-77 

NO 

WEATHER. STATIONS 

DR- 80 

NO 

ISR. DATA. REPORTS 

**** 

ISR. SATELLITES 

DR- 8  3 

NO 

I SR. PLANNING 

DR- 8  4 

NO 

ISR. TARGETS 

DR- 8  5 

NO 

ISR. SENSORS 

DR- 8  6 

NO 

ISR. EFFECTS 

DR-87 

NO 

ISR. INITIALIZATION 

DR- 8  8 

NO 

ISR. EVENTS 

DR- 8  9 

NO 

END . DATA . REPORT . CONTROLS 


OUTPUT . REPORT . CONTROLS 


COMPLETED. MIS SION. INFO 

AW-1 

NO 

BOTH 

0.00 

300.00 

DAILY . AIR . WAR . RESULTS . SUMMARIES 

•kic-kic 

MISSION . & . SORTIE . SUMMARY 

AW-2 

NO 

BOTH 

0.00 

300.00 

AIRCRAFT . LOSS . SUMMARY 

AW-3 

NO 

BOTH 

0.00 

300.00 

AIRBASE. ACTIVITY 

AW- 4 

NO 

BOTH 

0.00 

300.00 

SORTIE . GENERATION 

AW-5 

NO 

BOTH 

0.00 

300.00 

CANCELLED .MISSION . INFO 

AW- 6 

NO 

BOTH 

0.00 

300.00 

INDIVIDUAL . MISSION . KILL . RESULTS 

kkick 

DCA. MISSION. KILLS 

AW-7 

YES 

BOTH 

0.00 

300.00 
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ODCA. MISSION. KILLS 
HVAA. MISSION. KILLS 
BARCAP. MISSION. KILLS 
FSWP. MISSION. KILLS 
DTBM. MISSION. KILLS 
STI. MISSION. KILLS 
CAS. MISSION. KILLS 
BAI. MISSION. KILLS 
INT. MISSION. KILLS 
OCA. MISSION. KILLS 
DSEAD. MIS SION. KILLS 
SUPPRESSION .MISSION . KILLS 
OTBM. MISSION. KILLS 
AIR. ESCORT. MISSION. KILLS 
HIGH. VALUE .ASSET .ACTIVITY 
DEFENDING . AIRCRAFT . KILLS 
AIRCRAFT .MAINTENANCE 

TYPE . AIRCRAFT . MX . ACT I VITY 
CONSUM. RESOURCE .ACTIVITY 
NON . CONSUM . RES . ACTIVITY 
GROUND . COMBAT . CYCLE . REPORTS 
UNIT . STATUS 
C3 . FACILITY . STATUS 
CHOKE . POINT . STATUS 
FLOT. MOVEMENT 
ZONE . SECTOR . AD . STATUS 
TYPE . AD . SITE . STATUS 
STRATEGIC . TARGET . STATUS 
IADS. STATUS 

AIR .MISSION .PLANNING . REPORTS 
APPORTIONMENT .ALLOCATION 
AIR . TASKING . ORDERS 
SORTIES . SCHEDULED . SUMMARY 
TARGET . PRIORITIES 
AIR . REFUELING . ALLOCATION 
GROUND . SUPPLY . CYCLE . REPORTS 
LOGISTIC . FACILITY . STATUS 
SUPPLY . TRAIN . STATUS 
CRITICAL . RESOURCE . STATUS 
END . OUTPUT . REPORT . CONTROLS 

GAME . REPORT . SELECTION 
GC-1  NO  BOTH 
GC-2  NO  BOTH 
GC-3  NO  BOTH 
GC-4  NO  BOTH 
COMMANDS" 

GC-5 

COMMANDS" 

GC-6 
GC-7 
GC-8 
GC-9 
GC-10 
GC-11 
GC-12 
GC-1 3 


AW- 8 

YES 

BOTH 

0.00 

300.00 

AW- 9 

YES 

BOTH 

0.00 

300.00 

AW- 10 

YES 

BOTH 

0.00 

300.00 

AW- 11 

YES 

BOTH 

0.00 

300.00 

AW- 12 

NO 

BOTH 

0.00 

300.00 

AW- 13 

NO 

BOTH 

0.00 

300.00 

AW- 14 

NO 

BOTH 

0.00 

300.00 

AW-15 

NO 

BOTH 

0.00 

300.00 

AW- 16 

NO 

BOTH 

0.00 

300.00 

AW- 17 

NO 

BOTH 

0.00 

300.00 

AW- 18 

NO 

BOTH 

0.00 

300.00 

AW- 19 

NO 

BOTH 

0.00 

300.00 

AW-20 

NO 

BOTH 

0.00 

300.00 

AW-21 

YES 

BOTH 

0.00 

300.00 

AW-22 

NO 

BOTH 

0.00 

300.00 

AW-23 

YES 

icick-k 

BOTH 

0.00 

300.00 

MX-1 

NO 

BOTH 

0.00 

300.00 

MX-2 

NO 

BOTH 

0.00 

300.00 

MX- 3 

NO 

**** 

BOTH 

0.00 

300.00 

CC-1 

NO 

BOTH 

0.00 

300.00 

CC-2 

NO 

BOTH 

0.00 

300.00 

CC-3 

NO 

BOTH 

0.00 

300.00 

CC-4 

NO 

BOTH 

0.00 

300.00 

CC-5 

NO 

BOTH 

0.00 

300.00 

CC-6 

NO 

BOTH 

0.00 

300.00 

CC-7 

NO 

BOTH 

0.00 

300.00 

CC-8 

NO 

kkkk 

BOTH 

0.00 

300.00 

MP-1 

NO 

BOTH 

0.00 

300.00 

MP-2 

NO 

BOTH 

0.00 

300.00 

MP-3 

NO 

BOTH 

0.00 

300.00 

MP-4 

NO 

BOTH 

0.00 

300.00 

MP-5 

NO 

**** 

BOTH 

0.00 

300.00 

SC-1 

NO 

BOTH 

0.00 

300.00 

SC-2 

NO 

BOTH 

0.00 

300.00 

SC-3 

NO 

BOTH 

0.00 

300.00 

"AIR  COMMAND  MISSION  ALLOCATION  REPORT" 

"AIR  COMMAND  RESOURCE  ALLOCATION  REPORT" 
"TARGET  PRIORITY  REPORT" 

"GROUND  COMMAND  STRUCTURE  WITH  SUPPORTING  AIR 

NO  BOTH  "AIR  COMMAND  STRUCTURE  WITH  SUPPORTED  GROUND 

NO  BOTH  "AIR  BASE  ASSETS  REPORT" 

NO  BOTH  "LOGISTICS  FACILITY  ASSETS  REPORT" 

NO  BOTH  "AIRCRAFT  CHARACTERISTICS  REPORT" 

NO  BOTH  "ZONE  SECTOR  PERCEPTION  HISTORY" 

NO  BOTH  "FLOT  MOVEMENT  REPORT" 

NO  BOTH  "MISSION  ALLOCATION  PROJECTION" 

NO  BOTH  "PROJECTED  AIR  TASKING  ORDERS" 

NO  BOTH  "PROJECTED  SORTIES  SCHEDULED" 
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GC-14  NO  BOTH  "AIRCRAFT  PLANNING  FACTORS" 

GC-15  NO  BOTH  "PREPLANNED  ATO  REPORT" 

GC-16  NO  BOTH  "ZONE  SECTOR  AIR  DEFENSE  SITE  STATUS" 
GC-17  NO  BOTH  "AIR  COMBAT  RULES" 

GC-18  NO  BOTH  "AIR  PLANNING  RULES" 

END . GAME . REPORT . SELECTION 


TRANSACTION . CONTROLS 
@DEBUG . TRANSACTIONS 


ADFL 

0 

0.000 

300.00 

ADFH 

0 

0.000 

300.00 

AIRA 

0 

0.000 

300.00 

AIRF 

0 

0.000 

300.00 

AIRG 

0 

0.000 

300.00 

ANTC 

0 

0.000 

300.00 

ANTI 

0 

0.000 

300.00 

ANTP 

0 

0.000 

300.00 

BSEL 

0 

0.000 

300.00 

BSEM 

0 

0.000 

300.00 

BSEO 

0 

0.000 

300.00 

CONA 

0 

0.000 

300.00 

GRDC 

0 

0.000 

300.00 

GRDM 

0 

0.000 

300.00 

GRDO 

0 

0.000 

300.00 

ISRE 

0 

0.000 

300.00 

ISRP 

0 

0.000 

300.00 

ISRS 

0 

0.000 

300.00 

ISRT 

0 

0.000 

300.00 

LOGS 

0 

0.000 

300.00 

LOGE 

0 

0.000 

300.00 

NETM 

0 

0.000 

300.00 

NETP 

0 

0.000 

300.00 

PLAA 

0 

0.000 

300.00 

PLAT 

0 

0.000 

300.00 

RECT 

0 

0.000 

300.00 

RULP 

0 

0.000 

300.00 

STAA 

0 

0.000 

300.00 

UTLA 

0 

0.000 

300.00 

UTLG 

0 

0.000 

300.00 

UTLF 

0 

0.000 

300.00 

WTHH 

0 

0.000 

300.00 

@GRAPHIC . TRANSACTIONS 

GRAA 

3 

0.000 

300.00 

GRAG 

4 

0.000 

300.00 

GRAI 

0 

0.000 

300.00 

SANALYSIS.' 

TRANSACTIONS 

EQ. KILLS 

0  0. 

000  300.000 

AA. KILLS 

3  0. 

000  300.000 

SA. KILLS 

3  0. 

000  300.000 

MUNT 

.EXP 

3  0. 

000  300.000 

STR.TGT 

0  0. 

000  300.000 

AA.ENC 

0  0. 

000  300.000 

SDATABASE . 

TRANSACTIONS 

DB.PLAN 

0  0. 

000  300.000 

DB.EXEC 

0  0. 

000  300.000 

DB . STATE 

0  0. 

000  300.000 

@RMS . TRANSACTIONS 
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RMSPP  0  0.000  300.000 

@MP. TRANS ACT IONS 

MPPP  0  0.000  300.000 

@AIR . LIFT . TRANSACTIONS 

ALFT  0  0.000  300.000 

@ISR. TRANSACTIONS 

ISRPP  0  0.000  300.000 

END . TRANSACTION . CONTROLS 


RANDOM . NUMBER . STREAM . DATA 
RE AD. RANDOM. SEEDS 
WRITE . RANDOM . SEEDS 
S INGLE . SEED . ALL . NUMBERS 
SINGLE. SEED. IS (INT) 

RN. STREAMS 
ADFlOO.l  1 
ADF100.2  1 

ADF105.1  1 

ADF120.1  1 

ADF120.2  1 

ADF120.3  1 

ADF140.1  1 

ADF150.1  1 

ADF150.2  1 

ADF150.3  1 

ADF150.4  1 

ADF150.5  1 

ADF160.1  1 

ADF161.1  1 

ADF161.2  1 

ADF161.3  1 

ADF161.4  1 

ADF161.5  1 

ADF200.1  1 

ADF300.1  1 

ADF300.2  1 

ADF500.1  1 

AIR007.1  1 

AIR017.1  1 

AIR041.1  1 

AIR041.2  1 

AIR050.1  1 

AIR050.2  1 

AIR060.1  1 

AIR060.2  1 

AIR070.1  1 

AIR070.2  1 

AIR021.1  1 

AIRlOl.l  1 
AIR101.2  1 

AIRllO.l  1 
AIR120.1  1 

AIR150.1  1 

AIR150.2  1 

AIR241.1  1 

AIR526.1  1 


YES 

YES 

NO 

1 
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AIR528.1  1 
AIR540.1  1 
AIR550.1  1 
AIRS 61.1  1 
AIR561.2  1 
AIRS 64.1  1 
AIRS 67.1  1 
AIRS 67. 2  1 
AIR800.1  1 
AIR800.2  1 
AIR810.1  1 
AIR810.2  1 
AIR830.1  1 
AIR830.2  1 
AIR830.3  1 
AIR830.4  1 
AIR840.1  1 
AIR840.2  1 
AIR840.3  1 
AIR911.1  1 
AIR911.2  1 
AIR920.1  1 
BSE002.1  1 
BSE003.1  1 
BSE003.2  1 
BSEOSO.l  1 
BSE0S0.2  1 
BSE200.1  1 
BSE200.2  1 
BSE200.3  1 
BSE200.4  1 
BSE200.S  1 
BSE201.1  1 
BSE201.2  1 
BSE2S1.1  1 
BSE260.1  1 
BSE260.2  1 
BSE270.1  1 
BSE800.1  1 
GRDlOO.l  1 
GRD300.1  1 
GRD302.1  1 
GRD309.1  1 
GRD871.1  1 
GRD871.2  1 
ISROOO.l  1 
ISR000.2  1 
ISR000.3  1 
ISROOl.l  1 
ISRllS.l  1 
ISR11S.2  1 
ISR2S0.1  1 
ISR2S0.2  1 
ISR2S1.1  1 
ISR2S1.2  1 
ISR2S1.3  1 
ISR2S1.4  1 
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ISR360.1  1 
ISR365.1  1 
ISR400.1  1 
ISR500.1  1 
ISR500.2  1 
ISR500.3  1 
PLA443.1  1 
RECOll.l  1 
REC011.2  1 
REC011.3  1 
REC012.1  1 
REC012.2  1 
REC012.3  1 
REC012.4  1 
REC012.5  1 
REC012.6  1 
REC013.1  1 
REC013.2  1 
REC013.3  1 
REC013.4  1 
REC013.5  1 
REC013.6  1 
REC014.1  1 
REC014.2  1 
REC014.3  1 
REC014.4  1 
REC014.5  1 
REC014.6  1 
REC014.7  1 
REC014.8  1 
REC014.9  1 
REC015.1  1 
REC015.2  1 
REC015.3  1 
REC015.4  1 
REC015.5  1 
REC015.6  1 
REC015.7  1 
REC015.8  1 
REC016.1  1 
REC016.2  1 
REC016.3  1 
REC016.4  1 
REC016.5  1 
REC016.6  1 
REC016.7  1 
REC016.8  1 
REC016.9  1 
REC016.10  1 
REC016.il  1 
REC017.1  1 
REC017.2  1 
REC017.3  1 
REC017.4  1 
REC017.5  1 
REC017.6  1 
REC017.7  1 
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REC017 .8 

1 

REC017.9 

1 

REC017.10 

1 

REC018.1 

1 

REC018.2 

1 

REC018.3 

1 

REC018.4 

1 

REC310.1 

1 

REC310.2 

1 

REC320.1 

1 

REC320.2 

1 

REC320.3 

1 

REC330.1 

1 

REC330.2 

1 

REC330.3 

1 

REC330.4 

1 

REC330.5 

1 

REC330.6 

1 

REC330.7 

1 

REC330.8 

1 

REC341.1 

1 

REC341.2 

1 

REC900.1 

1 

UTL305.1 

1 

UTL520.1 

1 

UTL520.2 

1 

UTL550.1 

1 

UTL550.2 

1 

UTL750.1 

1 

UTL920.1 

1 

END. RN, STREAMS 

END . RANDOM . NUMBER . STREAM . DATA 


END . CONTROL 


airairpk.dat  -  F-15C, F-15C w/Radar, F-XX Scenarios 


AIR. AIR. PKS. 202 

BLUE. PK. MULTIPLIER (DEC)  1.00  RED. PK. MULTIPLIER (DEC)  1.00 

KILLERS 
BLUE 

@  KILL_ID  MUNT  - AIRCRAFT - 

10  102  1014  1015  1215  1016  1018  END 

11  103  1004  1026  1010  1014  1015  1215  1016  1018  END 

12  104  1004  1026  1014  1018  END 


12 

104 

1004 

1026 

1014 

RED 

KILL  ID 

MUNT 

-  AIRCRAFT - 

20 

201 

2001 

2021 

2023 

2029 

END 

21 

202 

2001 

2021 

2023 

2029 

END 

22 

203 

2001 

2021 

2029 

END 

23 

208 

2001 

2021 

2023 

2029 

END 

END. KILLERS 
PKS 

BLUE 


210  220  230 


Pk*s  qtall Blue  amt  Red  aircraft 
in  scenario  provided  by  ASCIXK 
reduced  to  Pk  ’s  in  bold  for  aU 
scenarios 


20  1  2  i 

21  2  2  i 

22  2  2  '* 

2i  2  2  1 

END . PKS 

LOW . RES . AIR . TO . AIR . PKS 
BLUE 

210 

1.000  25 

END . SET 

220 

1.000  25 

END . SET 

230 

1.000  25 

END . SET 

RED 

100 

1.000  2 

END . SET 

110 

1.000  5 

END . SET 

120 

1.000  5 

END . SET 
130 


Same  natation  as  abpve 


100 


1.000 

END . SET 

END. LOW. RES. PKS 
END. AIR. AIR. PKS 


5 
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airairpk.dat  -  All w/AIM-X  Scenarios 


AIR. AIR. PKS. 2 02 

BLUE. PK. MULTIPLIER (DEC)  1.00  RED.PK.MULTIPLIER(DEC)  1.00 


KILLERS 

BLUE 

0  KILL  ID 

MUNT 

- AIRCRAFT— 

10  101 

New  kiUertn  added  for  all  w/AIM-X  scenarios 

11 

102 

1014 

1215 

1016 

1018 

END 

12 

103 

1004  1026  1010 

1014 

1015 

1215 

1016 

1018 

END 

13 

104 

1004  1026 

1014 

1018 

END 

RED 

0  KILL  ID  MUNT  -  AIRCRAFT 


20 

201 

2001 

2021 

2023 

2029 

END 

21 

202 

2001 

2021 

2023 

2029 

END 

22 

203 

2001 

2021 

2029 

END 

23 

208 

2001 

2021 

2023 

2029 

END 

END . KILLERS 
PKS 

BLUE 

210  220  230 

10  6  7  8 

New  AISUX  Pts  adOedfor  all  AIM-^X  seemNos 

11  3  4  5 

12  3  4  7 

13  4  5  8 


RED 

100  110  120 
20  1  2  2 

21  2  2  2 

22  2  2  2 

23  2  2  2 

END . PKS 

LOW . RES . AIR . TO . AIR . PKS 
BLUE 

210 

1.000  25 

END . SET 

220 

1.000  25 

END . SET 

230 

1.000  25 

END . SET 

RED 

100 

1.000  2 

END . SET 

110 

1.000  5 

END . SET 

120 


130 

2 

2 

2 

2 
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1.000  5 

END . SET 

130 

1.000  5 

END . SET 

END. LOW. RES. PKS 
END. AIR. AIR. PKS 
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acserv.  dat  -  F-XX,  F-15C  w/AIM-X,  F-15C  w/Radar  and  AIM-x,  F-XX  w/AIM-X 

SERVICE. KITS. 304 


Win  ”F-15C  KW"  Change  “F-15C”  to '‘F-XX”  for  F-XX  scenarios 

SIDE. .NUM.AC 
1  24 

MUNITIONS. .ID. .NUM 

1^02  5000  Change  ”1102”  to  “I lOl” for  Am»X  in  alt  vhMM-X  scenarios 

1103  5000 

1104  5000 

REPAIR. RESOURCES. .ID. .NUM 

1  50 

2  25 

3  25 

INT . LEVEL . MAINTENANCE . FACS 
END. KIT 


END. SERVICE. KITS 


squadron.dat  -  F-XX  and  all  w/AIM-X  Scenarios 


SQUADRONS. 305 


10  T  5  "F-15C"  Change  “F.I5C"  to  ^‘F^XX"for  all  F^XX  scenarios 

DAY. IN. THEATER. .AUTH.QTY. SORT /DAY. .AC .MAX. SORT /DAY 
1.00  3.34  3.50 

6.00  2.75  3.34 

END. PROFILE 


@  INCIRLIK 

11601  "F-16_INCIRLIK" 

SIDE. .SUP.CMD.ID. .TYPE. AC. ID. .AUTH.QTY. .AR. PRIORITY. . .RECCE. PRIORITY 
1  1200  1016  37  1  0 

MOB. ID. .DISP.AB.ID. . SERV.KIT . ID . . SORT . PROF . ID . .MISSION . CLASS 

1003  1007  1016  1016  INCIRLIK. MULTI .ROLE 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . . INT . . .OCA. .OTBM 
. .DTBM 

0  70  70  70  70  70  50  70  70  70  90 

90  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

70  0000000000 

0  0  100 
ORDERS 
END . ORDERS 


1 1501  "F-  2  5C_INCIRLIK"  Change  "F-ISC”  to  '‘F-XX”  for  F-XX  scenaiios 
SIDE.  .SUP.CMb'VlD.  '.TYPE.  AC.  ID. ''.'AUTH.QTY.  .AR.  PM 

1  1200  1015  28  1  0 

MOB. ID. .DISP.AB.ID. . SERV . KIT . ID . . SORT . PROF . ID . .MISSION . CLASS 
1003  1004  1015  1015 

INCIRLIK . AIR . SUPERIORITY 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . .INT. . .OCA. .OTBM 
. .DTBM 

70  70  70  0  0  90  0  0  0  0  0 

0  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
..XXXX.. RESV 

00000000000 
0  0  70 

ORDERS 
END. ORDERS 


105 


@  TABUK 

•T~15CmBUK"  Change  ^‘F-^ISC^  to  •^F-Xr' for  an 

SIDE  .  .  sijP  .  CMD  .  ID  .  .  TYPE  .  AC .  ID  .  .  AUTH .  QTY .  .  AR .  PRIORITY .  .  .  RECCE  . PRIORITY 
1  1200  1015  24  1  0 

MOB. ID. .DISP.AB.ID. . SERV.KIT . ID . . SORT .PROF . ID . .MISSION. CLASS 
1004  1007  1015  1015  AIR. SUPERIORITY 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . .INT. . .OCA. .OTBM 
. .DTBM 

50  70  70  70  70  70  0  0  0  0  0 

0  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. . SREC . . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

0  0  0  0  0  0  0  0  0  0  0 

0  0  100 

ORDERS 

2.0  ARRIVE 
END . ORDERS 


@  AL-KHARJ 

11503  _AL~KHAPJ"  Change  ^*F-15C*  to ‘*F~XX*' for  all  F-XX  scenarios 

SIDE. .SUP. CMD. ID. .TYPE. AC. ID. .AUTH. QTY. .AR. PRIORITY. . .RECCE .PRIORITY 
1  1200  1015  24  1  0 

MOB. ID. .DISP.AB.ID. . SERV.KIT . ID . . SORT . PROF . ID . .MISSION. CLASS 
1007  1008  1015  1015  AIR. SUPERIORITY 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . . INT . . .OCA. .OTBM 
. .DTBM 

70  70  70  70  70  70  0  0  0  0  0 

0  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

00000000000 
0  0  100 

ORDERS 

3 . 0  ARRIVE 
END . ORDERS 


@  DHAHRAN 

1 1 504  ’^F-15C  DftANRAm  "  Change  “F-ISC’to  “F-XX^*  for  all  F'-XX scenarios 
sibE.'  .sup!  CM)'.  ib^.'TYPE^AC.  ID. ''.AUTh''’qTY.'’’.'aR. PRIORI^ 

1  1200  1015  24  1  0 

MOB. ID. .DISP.AB.ID. . SERV.KIT . ID . . SORT . PROF . ID . .MISSION. CLASS 
1009  1011  1015  1015  AIR. SUPERIORITY 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . .INT. . .OCA. .OTBM 
. .DTBM 

70  70  70  70  70  70  0  0  0  0  0 

0  0 
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•DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

0  0  0  0  0  0  0  0  0  0  0 
0  0  100 
ORDERS 

4.0  ARRIVE 
END. ORDERS 

1 1  ’}05  '’F-15C piiAHRAN?."  Change  “t'-l5CUo'*P~XX'' for atlP-XX scenarios 

SIDE. .SUP.CMD.ID. .TYPE. AC. ID. .AUTH.QTY. .AR. PRIORITY. . .RECCE. PRIORITY 
1  1200  1015  24  1  0 

MOB. ID. .DISP.AB.ID. . SERV.KIT . ID . . SORT . PROF . ID . .MISSION. CLASS 
1009  1011  1015  1015  AIR. SUPERIORITY 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . . INT . . .OCA. .OTBM 
. .DTBM 

70  70  70  70  70  70  0  0  0  0  0 

0  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

00000000000 
0  0  100 

ORDERS 

5.0  ARRIVE 
5.0  MERGE  11504 
END. ORDERS 


END . SQUADRONS 
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typeac.dat  - All  F-XX  scenarios  and  all  w/AIM-X  scenarios 

TYPE. AIR. CRAFT. 302 

NUMBER. OF. AIR. TO. AIR. CLASSES:  1 
10000  DEFAULT 
END . AIR . TO . AIR . CLASSES 
@ 

@  - 

@  DESERT  STORM  DATABASE 

@  - 

@  DATA  IN  THIS  FILE  FROM  UNCLASSIFIED  SOURCES 
@  IN  PARTICULAR,  PERFORMANCE  DATA,  RCS  ETC  TAKEN  DIRECTLY 
8  FROM  CACI  DISTIBUTION  TAPE  FOR  ME  &  DATASMALL  SCENARIO 
0  PARAMETERS  ARE  NOT  NECESSARILY  AUTHORITATIVE 

@  - 

0 


miMRER.  OF.  RADAR.  S IGNATURE .  OBJECTS:  IP.  Replace  "12”  with  '*13”  for  F-XX 

1010  "A-10" 

.RADAR. CROSS. SECTION(SQ.MTRS) . BY . RAD AR . BAND . . 
5.00 

i^ew  RCS  abject  added 

. RADAR . CROSS, SECT ION ($Q, FTRS) .BY.  RADAR, BAND , . 

scenarios  only 

1015  "F-15C"  Replace  “F-15C”  wRh  «F-XX‘’  for  F-XX  scenarios 


SIDE _ FIT. IN. SHELTER _ TGT. CLASS. . .TGT. INDEX. . . OCA. TGT. PRIORITY. .GRA 

PH 

1  1  "FIGHTER"  110  1 

2 

AIR. TO. AIR. CLASS 
10000 

RPV . DATA ( 0 . FOR . NON . RPV) 

RPV . TYPE (0,1,2)  0 

TBM. AIR. MUNITION. TYPE. ID  0 

PROB. ENEMY. INT. LAUNCH (0-1 00)  25 


ONBOARD . EQUIPMENT 


RADAR. ID 

ims  Replace  nmi”  wUh  »mrfor  F^^XXandMSC  yemdar 
scenarios 

TARGET . LOC . ID 

2001 

PERFORMANCE . DATA _ 

.  .ALTITUDE (METERS)  . . . 

SPEED (KNOTS) 

LOW. DASH 

200 

540 

LOW. PENETRATE 

200 

540 

HIGH. DASH 

8000 

540 

HIGH. PENETRATE 

8000 

540 

HIGH. CRUISE 

8000 

540 

CAP. ENEMY 

8000 

540 

CAP . FRIENDLY 

8000 

540 

BASE . RADAR , SIGNATORB . OBJECT , ID  1015 

all  F-XX 

Reptoce  mU” 
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TAKEOFF.LENGTH(METERS) .. .MISSION  900  DISPERSAL  900 

LANDING. LENGTH (METERS) .. .MISSION  900  NIGHT  900 

DAMAGED. RWY. FACTOR (METERS)  250 

SORTIE .GENERATION. DATA 

FLYING. PERIODS. . .DAY  3  NIGHT  2 
MISSION. DATA 

MIN. FLT. SIZE  1  ORBIT .WIDTH (MTRS)  50000  ORBIT .DEPTH (MTRS)  40000 
MAX. TARGETS. PER. SORTIE  1 

FLY.DIRECT(l=YES,2=NO)  1 

DURATION . OF . ALERT . WITHOUT . LAUNCH (MINUTES)  180 
PRIORITIES. FOR. SUPPORT. BY. . .EAIR. . .EJAM. . .ESUP. . .CORRIDOR 

111  1 
MISSION . EFFECTIVENESS . DATA ( 0-100) 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . . INT . . .OCA. .OTBM 
..DTBM 

50  75  75  80  75  75  0  0  0  0  0 

0  0 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. . SREC . . .AEW. . .AAR. .LIFT 
. .XXXX. .RESV 

00000000000 
0  0  100 

MISSION . ALTITUDE . DATA ( l=LOW, 2=HIGH) 

. .DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. . .STI. . .CAS. . .BAI. . .INT. . .OCA. .OTBM 
. .DTBM 

22222222222 
2  2 

.DSED. .SSUP. .CSUP. .ESUP. .SJAM. .CJAM. .EJAM. .RECC. .SREC. . .AEW. . .AAR. .LIFT 
. .XXXX 

22222222222 
2  2 

FUEL . CONFIGURATIONS 
3006  "38660  LBS.  FUEL  CAPACITY" 

CAPACITY (LBS) . .REFUELl (LBS) . . REFUEL2 (LBS) . .DELTA . RADAR . SIG . OBJ 
38660  3000  3000  0 

FUEL. BURN. PROFILES (LBS/MINUTE) 

4006  "FUEL  PROFILE  1" 


L . DASH . . L . PENETRT . . H . DASH . . H . PENETRT . . H . CRUISE . . CAP . ENEMY . . CAP . FRIEND 
10  10  10  10  10  100 


100 


AIR . TO . AIR . CONFIGURATIONS 

I  "4;  m^AAi4  i-  4:  A  CM- 9"  Reptad^^AMRAAM"  with for  all 

FUEL. CONFIG. .FUEL. BURN. PROF. .DELTA. RADAR. SIG. OBJ. .VULNERABILITY 

3006  4006  0  2 

LAUNCHE  S / AIR . ENG . . PCT . C2 . NO . AEW . . PCT . C2 . WI TH . AEW . . RECCE . SENSOR . ID 
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70 


95 


101 


2 

MISSION . EFFECTIVENESS 

. -DCA. .ODCA. .HVAA. .BARC. .FSWP. .EAIR. .DTBM 
75  75  75  75  75  75  0 

MUNITIONS. .ID. .NUMBER. .CEP (METERS) . . DELTA. RAD AR. SIG. OBJ 


iliiliilBi 

iiiigiiigii 

i||||lliigj 

mph^nor 

with 

ilH 

AIM-X 

BilBliHli 

smmrios 

BiliiiHP 

iilliilBii 

i03 

iiiiiiiiiiii 

iiiiiiiia 

iiiiiiiiiiii 
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iiiiiiiiiiii 

JAMMERS _ ID . 

. NUMBER 

END. CONFIG 

AIR . TO . GROUND . CONFIGURATIONS 

SPECIAL. CONF IGURAT IONS 

END . CONFIGURATIONS 
END. AIRCRAFT 


END. TYPE. AIRCRAFT 
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typerdr.dat  -  F-15Cw/Radar  and  all  F-XX  Scenarios 


TYPE. RADAR. 101 

NUMBER. OF. RADAR. BANDS:  1 
ID. . . .NAME 

10001  "RADAR  BAND  1" 
END. RADAR. BANDS 


NUMBER.  OF.  R^AR.  DETECTION.  OBJECTS:  21 

Additional  rai^r  object  added  for  P-ISC 

wiRadarmd  aH  P-XX  scenaHm 

1001  "BLUE  AD  FC  RADAR" 

SIDE 

1 

BAND 

10001 

SWEEP. ANGLE (DEG) 

60 

1. SQ.MTR. RANGE (M) 

95000 

1 . SQ . MTR . ALTITUDE (M) 

25000 

MAX . RANGE (M) 

50000 

1002  "BLUE  AD  ACQ  RADAR" 

SIDE 

1 

BAND 

10001 

SWEEP. ANGLE (DEG) 

360 

1. SQ.MTR. RANGE (M) 

146000 

1. SQ.MTR. ALTITUDE (M) 

25000 

MAX. RANGE (M) 

100000 

1003  "BLUE  AC  RADAR" 

SIDE 

1 

BAND 

10001 

SWEEP. ANGLE (DEG) 

90 

1. SQ.MTR. RANGE (M) 

120000 

1. SQ.MTR. ALTITUDE (M) 

120000 

MAX . RANGE (M) 

250000 

t00[^  "RhVK  Ai.  radar  -  dual 

Ne)r  radar  added  for  P^tSC  wtRadar  and 

iiiiiiiiiiii 

F-XX  scenarios 

10001 

SNEER . ANGLE (DEG) 

120 

X.  SQ.MTR.  RANGE  (M) 

240000 

1 . SQ . MTR . ALT  TTUDE (M) 

120000 

MAX.  RANGE  (M) 

500000 

END . GROUND . TYPE . RADARS 

NUMBER. OF. AIR. TYRE. RADARS: 

3  Number  of  type  radars  changed from  "2”  to  ‘*3'’ for 

f\ISC 

wt Radar  and  aU  P>XK  scenarios 

1003  "BLUE  AC  RADAR" 

SIDE  1 

DETECT. OBJ. ID  1003 
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1005  ''BLUE  lie  PAJUAR  -  BUAl"  N6W  radar  oddedfor R~t5C  wlHodar and 

BLUE  1  F-XXsmtarios 

DETECT. OBJ. ID  100b 


2003  "RED  AC  RADAR" 

SIDE  2 

DETECT. OBJ. ID  2003 

END . AIR . TYPE . RADARS 

END . TYPE . RADARS 
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typejam.dat  -  F-15Cw/Radar  and  all  F-XX  Scenarios 


TYPE. JAMMERS. 102 


BadattWS  added  for  F- ISC  w/Radar  and  alt  F-XX  scenarios 

RBL>.  jmM.Kk  .IDS _ CURVES . .  f  x  ^-RAtiGE,  TO.  JAmKR _ y-irt,  BVRimm 

2001  1  1200  79  13891  283  22b82  617  33271  1084  43964  1696 

2484  6534b  350/  76036  4900  86727  '7010  97418  11115 
39499  118800  END. CURVE 

2002  1  1200  103  11891  372  22582  8W  33273  1422  43964  2227 

3260  65345  4603  76036  6432  86727  9201  97418  14589 

iiiiiliiiliiiiiiiiiiiii^^^^ 

51843  118800  END. CURVE 

END. FED .  JAEmER . DATA 


END . RED . JAMMER . DATA 

1005  Radar  1005  added  for  F-JSC  wf Radar  and 

FED.JAmiER .  IDS _ (Im.  BURNTHRU)  scenarios 

END. RED.  JAEINER.  DATA 


END . BLUE . RADAR . DETECT ION . OBJ . DATA 
END . TYPE . JAMMERS 
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detect  .dat  -  F-15C  w/Eiadar  and  all  F-XX  Scenarios 


DETECT. PROBS. 205 


MULT . FACTORS . , 

.  .BLUE. 

.  .RED 

NO.AEW 

1.00 

1.00 

BLUE.AEW 

1.00 

1.00 

RED.AEW 

1.00 

1.00 

BOTH.AEW 

1.00 

1.00 

BLUE. KILLER. AC 

2001  2006  2021  2023 

2025  2029 


END 

.RED. 

TGT.AC 

.  . .PROB. 

DETECT , 

.  IF. 

.NO. 

AEW. 

.BLUE 

.AEW. 

.RED 

.AEW. 

.BOTH 

.AEW 

1010 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

1099 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1016 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

40 

50 

30 

40 

1004 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

1011 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

40 

50 

30 

40 

1211 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

iiiiii 

95 

99 

76 

95 

95 

99 

76 

95 

95 

99 

76 

95 

95 

99 

7$ 

^5 

95 

99 

76 

95 

70 

93 

58 

70 

Detection  probabilities  changed  to 

yaims 

tnboMforR^tSC 

w/RadarandallF-XX 

scenanos 

1215 

75 

94 

56 

. 75 . 

75" 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

15 

50 

63 

38 

50 

1017 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1052 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

1008 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

20 

25 

15 

20 

1018 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

75 

50 

63 

38 

50 

1006 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 
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0 

0 

0 

0 

0 

0 

0 

0 

1026 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

1007 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

1014 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56 

75 

75 

94 

56  75 

75 

94 

56 

75 

50 

63 

38 

50 

1003 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

1098 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

RED. KILLER, 

■  AC 

1010 

1099 

1016 

1004 

1011 

1211 

1015 

1215 

1017 

1052 

1008 

1018 

1006 

1026 

1007 

1014 

1003 

1098 

END. BLUE. TGT. AC. . . PROB .DETECT . IF . .NO.AEW. .BLUE.AEW. .RED.AEW. .BOTH.AEW 


2001 

60 

75 

45 

60 

12 

12 

12 

12 

60 

75 

45 

60 

60 

75 

45  60 

60 

75 

4$ 

60 

60 

73 

4$ 

60 

14 

14 

14 

14 

60 

75  45  60 

XX  scenarios 

Detection  probabilities  changed  to  values  in  bold  for  all  F- 

14 

14 

14 

14 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45  60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45  60 

60 

75 

45 

60 

60 

75 

45 

60 

2006 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0  0 

0 

0 

0 

0 

0 

0 

0 

0 

2021 

60 

75 

45 

60 

12 

12 

12 

12 

60 

75 

45 

60 

60 

75 

45  60 

60 

75 

45 

60 

60 

75 

45 

60 

14 

14  14 

14 

60 

75  45  60  DeKcliottpeobabliities  changed  to  values  in  bold  for  all  t'- 

XX  scenarios 


14  14  14  14  60  75  45  60  60  75  45  60  60 

75  45  60 
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75  45  60 


2023 

75  45  60 

75  45  £0 

XXscenarioK 

75  45  60 

75  45  60 

2025 
0  0  0 

0  0  0 

0  0  0 

0  0  0 

2029 

75  45  60 

75  45  60 

XX  scenarios 

75  45  60 
75  45  60 
END. DETECT 


60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

12 

12 

12 

12 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

14 

14 

14 

14 

60 

Detection  probabilities  changed  to  values  in  bold  for  all  /<’- 


14 

14 

14 

14 

60 

75 

45 

60 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

60 

75 

45 

60 

12 

12 

12 

12 

60 

75 

45 

60 

60 

60 

mm 

45 

60 

60 

iiii 

45 

60 

14 

14 

14 

14 

60 

Detection  probabilities  changed  to 

values  in 

bold  for  all  P- 

14 

14 

14 

14 

60 

75 

45 

60 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

75 

45 

60 

60 

60 

75 

45 

60 

60 

75 

45 

60 

PROBS 
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critres.dat  -  All  w/AIM-X  Scenarios 


CRITICAL . RESOURCES .380 

NUMBER. OF. CRITICAL. RESOURCES  39 


nOl  AIM-V  ID  1101  chaaged  fnyai  "AIM-?"  to  for  all  AIM-X 

TYPE. . . (1=AIR.MUNT,2=SA.MUNT,3=EQUIP)  1 

URGENCY. CURVE  1 

LOW. RES. INITIAL. STOCK  0 

LOW. RES. RESUPPLY. . . START .TIME (DEC. DAYS) . . .NUMBER/DAY 

HI.  RES.  INITIAL.  STOCKS . LOG.  FAC.  ID.  .  .NUMBER 

1006  250 

HI. RES. RESUPPLY. . .TIME (DAYS) . . .NUMBER. . . TARGET . TYPE . . .ID. LIST 
END. RE SOURCE 


END . CRITICAL . RESOURCES 
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relrngadvn.dat  -  All  w/AIM-X  Scenarios 


RRA.204 

BLUE 


20 

21 

22 

23 

10 

|lli 

1 

llii 

i  Hm  mapon  ID  and  rat^o  advantages  addodfordd  wfMM*K 

scenarios 

11 

1 

1 

1 

1 

12 

0 

0 

0 

1 

13 

0 

0 

0 

0 

RED 

10 

11 

12 

1 2  New  weapon  W  andmnge  advantages  (in  hotd)  added  for  cM  wfAM^ 

liiiiliii 

0 

0 

a 

1  scenarios 

iiiiiiiili 

0 

0 

0 

22 

0 

0 

1 

iHiilii; 

0 

0 

0 

END.RRA 

airmunit.dat  -  All  w/AIM-X  Scenarios 

TYPE. AIR 

.MUNITIONS. 301 

NUMBER. OF. AIR. MUNITION. TYPES  25 

. . .ID. .NAME. 

•  •  •  • 

•  «  «  . 

iliiiiiiliiiii 

2  AA  2  SCO  1 

m  not  changed  from  to  ^AIM^ryoratl  wlAIM^X  scenarios 

1102 

"AMRAAM" 

1 

AA 

2 

400 

1 

1103 

"AIM-9" 

1 

AA 

2 

200 

1 

1104 

"20MM  CANNON" 

1 

AA 

2 

50 

1 

1105 

"MK-82" 

1 

AG 

2 

500 

1 

1106 

"AGM-65" 

1 

AG 

2 

500 

1 

1107 

"AGM-88" 

1 

HARM 

2 

500 

1 

1108 

"SPW-45" 

1 

SPW 

2 

500 

1 

1109 

"B-DELAY  MINE" 

1 

MINE 

2 

500 

1 

1110 

"B-LETHAL  MINE" 

1 

MINE 

2 

500 

1 

1111 

"CBU-87  CEM" 

1 

AG 

2 

200 

2 

1112 

"CBU-97  SFW" 

1 

AG 

2 

200 

2 

1113 

"LGB  GBU" 

1 

AG 

2 

500 

1 

1114 

"GPS  ALL  WX  GBU" 

1 

AG 

2 

500 

1 

2201 

"AA-7" 

2 

AA 

2 

500 

1 

2202 

"AA-8" 

2 

AA 

2 

500 

1 

2203 

"AA-10" 

2 

AA 

2 

500 

1 

2204 

"FAB-250" 

2 

AG 

2 

500 

1 

2205 

"AS-10" 

2 

AG 

2 

500 

1 

2206 

"HARM- RED" 

2 

HARM 

2 

500 

1 

2207 

"SPW-RED" 

2 

SPW 

2 

500 

1 

2208 

"GSH-23" 

2 

AA 

2 

500 

1 

2209 

"R-DELAY  MINE" 

2 

MINE 

2 

500 

1 

2210 

"R-LETHAL  MINE" 

2 

MINE 

2 

500 

1 

2211 

"RBK-500  AT" 

2 

AG 

2 

500 

1 

END. AIR. MUNITION. TYPES 
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NUMBER . OF . AIR . MUNITION . STICK . TYPES  25 

.  .  .ID.  .NAME . TYPE.NUM. MULT. RAD.  .NUM.STDOFF. 

PCT 


.  •  . 

.  •  •  • 

•  •••••••••  »  ■ 

WPN. 

KILL 

.EFF. 

SUBM. 

.RANGE, 

.FC 

iiii 

not 

2 

0 

iiii» 

iliiiii: 

0 

lO  10 1  changed  from  ‘Vl/M-7”  to  **AlM-K**for  all  wfAlM-X  \cenafios 

102 

"AMRAAM" 

1102 

1 

2 

0 

0 

0 

0 

103 

"AIM-9” 

1103 

1 

2 

0 

0 

0 

0 

104 

"20MM  CANNON" 

1104 

1 

2 

0 

0 

0 

0 

105 

"6  MK-82" 

1105 

6 

1 

1 

0 

1000 

0 

106 

"AGM-65" 

1106 

1 

2 

1 

0 

5000 

0 

107 

"AGM-88" 

1107 

1 

2 

1 

0 

10000 

50 

108 

"SPW-45" 

1108 

1 

2 

1 

0 

10000 

50 

109 

"B-DELAY  MINE" 

1109 

4 

2 

1 

200 

1000 

0 

110 

"B-LETHAL  MINE" 

1110 

4 

2 

1 

200 

1000 

0 

111 

"4  CBU-87" 

1111 

4 

1 

1 

0 

1000 

0 

112 

"6  CBU-97" 

1112 

6 

1 

1 

0 

1000 

0 

113 

"LGB  GBU" 

1113 

1 

2 

1 

0 

10000 

0 

114 

"GPS  GBU" 

1114 

1 

2 

1 

0 

10000 

0 

201 

"AA-7" 

2201 

1 

2 

0 

0 

0 

0 

202 

"AA-8" 

2202 

1 

2 

0 

0 

0 

0 

203 

"AA-10" 

2203 

1 

2 

0 

0 

0 

0 

204 

"6  FAB-250" 

2204 

6 

1 

1 

0 

1000 

0 

205 

"AS-10" 

2205 

1 

2 

1 

0 

10000 

0 

206 

"HARM-RED" 

2206 

1 

2 

1 

0 

10000 

50 

207 

"SPW-RED" 

2207 

1 

2 

1 

0 

10000 

50 

208 

"GSH-23" 

2208 

1 

2 

1 

0 

1000 

0 

209 

"R-DELAY  MINE" 

2209 

4 

2 

1 

200 

1000 

0 

210 

"R-LETHAL  MINE" 

2210 

4 

2 

1 

200 

1000 

119 


211  "6  RBK-500" 


2211 


6  1 


1 


0  1000 


0 

END. AIR. MUNITION. STICK. TYPES 
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Appendix  E,  Estimated  Half  Length  for  90%  Interval 


The  table  below  lists  the  estimated  90%  h.l.  for  each  of  the  output  responses  for  31 
replications  of  the  simulation  based  on  the  variance  of  the  first  four  replications  of  each 
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4 

4 
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0.0068 
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AirSup 
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BlueSA 
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0.0040 
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0.0002 

0.0002 

0.0002 

AirSup 
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BlueSA 
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BlueGmd 

3 

5 

2 
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RedTEL 

73 
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RedFC 

96 
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alternative  solution. 
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Total  Red  Total  Total  Red  Total 

Red  Blue  Air  Total  Blue  Destroyed  Red  Acquistion  Red  FC 

RedA-A  Percent  Loss  Supremacy  BlueS-A  Destroyed  in  the  TELs  Radars  Radars 
Alternative  Run  Losses  Sortie  /Sortie  (.10)  Losses  on  Ground  Open  Killed  Killed  Killed 

F-15C  31  188  0.0371  0.0026  25  0.005109  107  245  6095  117  4876 
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Total  Red  Total  Total  Red  Total 

Red  Blue  Air  Total  Blue  Destroyed  Red  Acquistlon  Red  FC 

Red  A-A  Percent  Loss  Supremacy  Blue  S-A  Destroyed  in  the  TELs  Radars  Radars 

Alternative  Run  Losses  Sortie  /Sortie  (.10)  Losses  on  Ground  Open  Killed  Killed  Killed 
F-15CRdr  5  202  0.0283  0.0027  24  0.004932  77  255  5974  10  5059 
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Red  Blue  Air  Total  Blue  Destroyed  Red  Acquistion  Red  FC 

Red  A-A  Percent  Loss  Supremacy  Blue  S-A  Destroyed  in  the  TELs  Radars  Radars 
Alternative  Run  Losses  Sortie  /Sortie  (.10)  Losses  on  Ground  Open  Kilied  Kiiied  Kiiied 
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Total  Red  Total  Total  Red  Total 

Red  Blue  Air  Total  Blue  Destroyed  Red  Acquistion  Red  FC 

Red  A-A  Percent  Loss  Supremacy  Blue  S-A  Destroyed  in  the  TELs  Radars  Radars 
Alternative  Run  Losses  Sortie  /Sortie  (.10)  Losses  on  Ground  Open  Killed  Killed  Killed 

F-XXMsI  19  315  0.0005  0.0014  17  0.004345  51  177  6474  37  5797 
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Appendix  G,  Response  Distributions 


•JMP  histograms  below  for  the  output  responses  listed  in  Appendk  F  are  from  31 
replications  of  THUNDER  6.4.2  for  six  alternative  aircraft/weapons  combinations. 
Normal  curves  superimposed  on  the  graphs  are  constructed  from  the  mean  and  standard 
deviation  of  the  column. 
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Appendix  H,  90%  Bonferroni  Confidence  Intervals  for  Output  Responses 
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Appendix  I,  Centroid  Clustering  Data 


The  first  column  of  each  alternative  lists  the  cluster  membership  for  centroid  linkage 
under  unconstrained  conditions.  The  second  column  under  each  alternative  lists  the 
cluster  membership  for  centroid  linkage  under  constraints  of  six  total  clusters. 
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Appendix  J,  Two  and  Four  Factor  Rotations 


2  Factor  Rotation 
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-0.29569 


-0.33423 


-0.13976 


0.785061 


lEMMSl 


0.111223 


-0.06696 


-0.01197 


0.448551 


0.453344 


0.105828 


-0.98571 


-1.4176 


-1.04477 


-1.29407 


-1.00092 


-1.11026 


Factor2 


-0.17351 


1.23893 


2.109338 


0.357848 


-0.99126 


-0.55558 


-1.02489 


0.519494 


0.174084 


-1.58682 


0.919755 


0.02598 


0.361228 


2.528429 


0.460235 


-0.19639 


0.176109 


2.196229 


-0.4208 


-0.68293 


0.381024 


0.400459 


0.787326 


0.071208 


-0.70127 


0.909589 


0.722447 


-1.21101 


2.353894 


-0.39558 


1.948727 


0.100158 


0.368379 


1.038164 


-0.35684 


0.283747 


-1.15963 


-1.30056 


-1.13954 


-0.90403 


-0.72934 


-0.44977 


-1.26183 


-0.95722 


-1.03551 


-1.58862 


-0.88041 


-0.64916 


-1.88664 


0.297044 


0.822195 


0.60478 


-2.04588 


-1.43111 


ElHi 

msEm 


0.962461 


1.87976 


0.632722 


Factor3 


-1.00208 


1.441485 


0.682831 


1.074034 


-1.62246 


-1.46993 


-1.17989 


-0.77257 


1.411805 


-0.36597 


0.454954 


-0.69834 


1.135017 


1.676029 


-1.09868 


0.244736 


0.710755 


0.050586 


-0.19938 


-0.48238 


-0.45508 


1.423219 


-1.20778 


-0.47456 


-0.38283 


-0.66314 


1.901657 


-2.01881 


0.26223 


-0.57275 


0.153998 


-1.09175 


-0.72405 


0.622006 


0.558653 


1.017317 


-1.37896 


1.115624 


-1.01811 


0.224057 


1.151381 


-1.2066 


0.484782 


-0.51937 


0.303743 


0.760841 


-1.76821 


-1.26323 


Appendix  L,  MineSet  Schema 


Schema  for  Output  Response  Visualization 

# 

# 

# 

input  { 

#  Data  input  file 

# 

file  "newSmoe.data"; 

string  alternative; 
string  run; 
int  redloss; 
float  redpercnt; 
float  blosssort; 
float  airsup; 
float  bluesa; 
int  bluegrnd; 
int  redopen; 
int  redfc; 

} 


Schema  for  Factor  Score  Visualization 

# 

# 

# 

input  { 

#  Data  input  file 

# 

file  "invcosts.data"; 

string  alternative; 
string  run; 
int  redloss; 
float  redpercnt; 
float  blosssort; 
float  airsup; 
float  bluesa; 
int  bluegrnd; 
int  redopen; 
int  redfc; 
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float  factor  1; 
float  factor!; 
float  factor3; 
float  factorl!; 
float  factor!!; 
float  factorlll; 
float  invfacl; 
float  invfac!; 
float  invfacS; 
float  invfacl!; 
float  invfac!!; 
float  invfacl  11; 
float  LCC; 
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